research-article

Surgment: Segmentation-enabled Semantic Search and Creation of Visual Question and Feedback to Support Video-Based Surgery Learning

Authors:

Tandis Soltani,

Xu WangAuthors Info & Claims

CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems

Article No.: 461, Pages 1 - 18

https://doi.org/10.1145/3613904.3642587

Published: 11 May 2024 Publication History

Abstract

Videos are prominent learning materials to prepare surgical trainees before they enter the operating room (OR). In this work, we explore techniques to enrich the video-based surgery learning experience. We propose Surgment, a system that helps expert surgeons create exercises with feedback based on surgery recordings. Surgment is powered by a few-shot-learning-based pipeline (SegGPT+SAM) to segment surgery scenes, achieving an accuracy of 92%. The segmentation pipeline enables functionalities to create visual questions and feedback desired by surgeons from a formative study. Surgment enables surgeons to 1) retrieve frames of interest through sketches, and 2) design exercises that target specific anatomical components and offer visual feedback. In an evaluation study with 11 surgeons, participants applauded the search-by-sketch approach for identifying frames of interest and found the resulting image-based questions and feedback to be of high educational value.

Supplemental Material

MP4 File - Video Preview

Video Preview

Download
32.87 MB

MP4 File - Video Presentation

Video Presentation

Transcript for: Video Presentation

MP4 File - Video Figure

a video figure within 5 min.

Transcript for: Video Figure

References

[1]

Jad M. Abdelsattar, T.K. Pandian, Eric J. Finnesgard, Moustafa M. El Khatib, Phillip G. Rowse, EeeLN H. Buckarma, Becca L. Gas, Stephanie F. Heller, and David R. Farley. 2015. Do You See What I See? How We Use Video as an Adjunct to General Surgery Resident Education. Journal of Surgical Education 72, 6 (2015), e145–e150. https://doi.org/10.1016/j.jsurg.2015.07.012

[2]

Akgul Ahmet, Kus Gamze, Mustafaoglu Rustem, and Karaborklu Argut Sezen. 2018. Is Video-Based Education an Effective Method in Surgical Education? A Systematic Review. Journal of Surgical Education 75, 5 (2018), 1150–1158. https://doi.org/10.1016/j.jsurg.2018.01.014

[3]

Knut Magne Augestad, Khayam Butt, Dejan Ignjatovic, Deborah S Keller, and Ravi Kiran. 2020. Video-based coaching in surgical education: a systematic review and meta-analysis. Surgical endoscopy 34 (2020), 521–535.

[4]

Knut Magne Augestad, Khayam Butt, Dejan Ignjatovic, Deborah S. Keller, and Ravi Kiran. 2020. Video-based coaching in surgical education: a systematic review and meta-analysis. Surgical Endoscopy 34, 2 (01 Feb 2020), 521–535. https://doi.org/10.1007/s00464-019-07265-0

[5]

Ignacio Avellino, Sheida Nozari, Geoffroy Canlorbe, and Yvonne Jansen. 2021. Surgical Video Summarization: Multifarious Uses, Summarization Process and Ad-Hoc Coordination. Proc. ACM Hum.-Comput. Interact. 5, CSCW1, Article 140 (apr 2021), 23 pages. https://doi.org/10.1145/3449214

Digital Library

[6]

Ayan Kumar Bhunia, Yongxin Yang, Timothy M. Hospedales, Tao Xiang, and Yi-Zhe Song. 2020. Sketch Less for More: On-the-Fly Fine-Grained Sketch-Based Image Retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]

Carrie J. Cai, Emily Reif, Narayan Hegde, Jason Hipp, Been Kim, Daniel Smilkov, Martin Wattenberg, Fernanda Viegas, Greg S. Corrado, Martin C. Stumpe, and Michael Terry. 2019. Human-Centered Tools for Coping with Imperfect Algorithms During Medical Decision-Making. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3290605.3300234

Digital Library

[8]

Michelene TH Chi and Ruth Wylie. 2014. The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational psychologist 49, 4 (2014), 219–243.

[9]

ClipDrop. 2022. Cleanup.pictures - Remove objects, people, text and defects from any picture for free. https://cleanup.pictures/

[10]

M. Cooper and J. Foote. 2005. Discriminative techniques for keyframe selection. In 2005 IEEE International Conference on Multimedia and Expo. 4 pp.–. https://doi.org/10.1109/ICME.2005.1521470

[11]

Tobias Czempiel, Magdalini Paschali, Daniel Ostler, Seong Tae Kim, Benjamin Busam, and Nassir Navab. 2021. OperA: Attention-Regularized Transformers for Surgical Phase Recognition. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2021, Marleen de Bruijne, Philippe C. Cattin, Stéphane Cotin, Nicolas Padoy, Stefanie Speidel, Yefeng Zheng, and Caroline Essert (Eds.). Springer International Publishing, Cham, 604–614.

Digital Library

[12]

Ryan Daniel, Tyler McKechnie, Colin C Kruse, Marc Levin, Yung Lee, Aristithes G Doumouras, Dennis Hong, and Cagla Eskicioglu. 2022. Video-based coaching for surgical residents: a systematic review and meta-analysis. Surgical Endoscopy (2022), 1–11.

[13]

Bidyut Das, Mukta Majumder, Santanu Phadikar, and Arif Ahmed Sekh. 2021. Automatic question generation and answer assessment: a survey. Research and Practice in Technology Enhanced Learning 16, 1 (18 Mar 2021), 5. https://doi.org/10.1186/s41039-021-00151-1

[14]

Rashmi Datta, KK Upadhyay, and CN Jaideep. 2012. Simulation and its role in medical education. Medical Journal Armed Forces India 68, 2 (2012), 167–172.

[15]

AIKATERINI DEDEILIA, MARINOS G. SOTIROPOULOS, JOHN GERRARD HANRAHAN, DEEPA JANGA, PANAGIOTIS DEDEILIAS, and MICHAIL SIDERIS. 2020. Medical and Surgical Education Challenges and Innovations in the COVID-19 Era: A Systematic Review. In Vivo 34, 3 suppl (2020), 1603–1611. https://doi.org/10.21873/invivo.11950

[16]

Sounak Dey, Pau Riba, Anjan Dutta, Josep Llados, and Yi-Zhe Song. 2019. Doodle to Search: Practical Zero-Shot Sketch-Based Image Retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]

Andrew C Esposito, Nathan A Coppersmith, Erin M White, and Peter S Yoo. 2022. Video coaching in surgical education: utility, opportunities, and barriers to implementation. Journal of Surgical Education 79, 3 (2022), 717–724.

[18]

Facebook. 2022. React.js. https://github.com/fabricjs/fabric.js.

[19]

Zhihao Fan, Zhongyu Wei, Piji Li, Yanyan Lan, and Xuanjing Huang. 2018. A Question Type Driven Framework to Diversify Visual Question Generation. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18. International Joint Conferences on Artificial Intelligence Organization, 4048–4054. https://doi.org/10.24963/ijcai.2018/563

[20]

Firebase. 2023. Firebase JavaScript SDK. https://github.com/firebase/firebase-js-sdk.

[21]

C. Ailie Fraser, Joy O. Kim, Hijung Valentina Shin, Joel Brandt, and Mira Dontcheva. 2020. Temporal Segmentation of Creative Live Streams. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376437

Digital Library

[22]

Carly R Garrow, Karl-Friedrich Kowalewski, Linhong Li, Martin Wagner, Mona W Schmidt, Sandy Engelhardt, Daniel A Hashimoto, Hannes G Kenngott, Sebastian Bodenstedt, Stefanie Speidel, Beat P Müller-Stich, and Felix Nickel. 2021. Machine Learning for Surgical Phase Recognition: A Systematic Review. Annals of surgery 273, 4 (April 2021), 684—693. https://doi.org/10.1097/sla.0000000000004425

[23]

Tomer Golany, Amit Aides, Daniel Freedman, Nadav Rabani, Yun Liu, Ehud Rivlin, Greg S. Corrado, Yossi Matias, Wisam Khoury, Hanoch Kashtan, and Petachia Reissman. 2022. Artificial intelligence for phase recognition in complex laparoscopic cholecystectomy. Surgical Endoscopy 36, 12 (01 Dec 2022), 9215–9223. https://doi.org/10.1007/s00464-022-09405-5

[24]

Google. 2023. Jamboard. https://jamboard.google.com/

[25]

Tovi Grossman, Justin Matejka, and George Fitzmaurice. 2010. Chronicle: capture, exploration, and playback of document workflow histories. In Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology (New York, New York, USA) (UIST ’10). Association for Computing Machinery, New York, NY, USA, 143–152. https://doi.org/10.1145/1866029.1866054

Digital Library

[26]

Hongyan Gu, Yuan Liang, Yifan Xu, Christopher Kazu Williams, Shino Magaki, Negar Khanlou, Harry Vinters, Zesheng Chen, Shuo Ni, Chunxu Yang, 2023. Improving Workflow Integration with xPath: Design and Evaluation of a Human-AI Diagnosis System in Pathology. ACM Transactions on Computer-Human Interaction 30, 2 (2023), 1–37.

Digital Library

[27]

Hongyan Gu, Chunxu Yang, Mohammad Haeri, Jing Wang, Shirley Tang, Wenzhong Yan, Shujin He, Christopher Kazu Williams, Shino Magaki, and Xiang ’Anthony’ Chen. 2023. Augmenting Pathologists with NaviPath: Design and Evaluation of a Human-AI Collaborative Navigation System. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 349, 19 pages. https://doi.org/10.1145/3544548.3580694

Digital Library

[28]

Annetje CP Guédon, Senna EP Meij, Karim NMMH Osman, Helena A Kloosterman, Karlijn J van Stralen, Matthijs CM Grimbergen, Quirijn AJ Eijsbouts, John J van den Dobbelsteen, and Andru P Twinanda. 2021. Deep learning for surgical phase recognition using endoscopic videos. Surgical endoscopy 35 (2021), 6150–6157.

[29]

Narayan Hegde, Jason D. Hipp, Yun Liu, Michael Emmert-Buck, Emily Reif, Daniel Smilkov, Michael Terry, Carrie J. Cai, Mahul B. Amin, Craig H. Mermel, Phil Q. Nelson, Lily H. Peng, Greg S. Corrado, and Martin C. Stumpe. 2019. Similar image search for histopathology: SMILY. npj Digital Medicine 2, 1 (21 Jun 2019), 56. https://doi.org/10.1038/s41746-019-0131-z

[30]

Catherine M. Hicks, Vineet Pandey, C. Ailie Fraser, and Scott Klemmer. 2016. Framing Feedback: Choosing Review Environment Features That Support High Quality Peer Assessment(CHI ’16). Association for Computing Machinery, New York, NY, USA, 458–469. https://doi.org/10.1145/2858036.2858195

Digital Library

[31]

W. Y. Hong, C. L. Kao, Y. H. Kuo, J. R. Wang, W. L. Chang, and C. S. Shih. 2020. CholecSeg8k: A Semantic Segmentation Dataset for Laparoscopic Cholecystectomy Based on Cholec80. arxiv:2012.12453 [cs.CV]

[32]

Yue-Yung Hu, Laura M. Mazer, Steven J. Yule, Alexander F. Arriaga, Caprice C. Greenberg, Stuart R. Lipsitz, Atul A. Gawande, and Douglas S. Smink. 2017. Complementing Operating Room Teaching With Video-Based Coaching. JAMA Surgery 152, 4 (04 2017), 318–325. https://doi.org/10.1001/jamasurg.2016.4619

[33]

David A. Joyner, Wade Ashby, Liam Irish, Yeeling Lam, Jacob Langston, Isabel Lupiani, Mike Lustig, Paige Pettoruto, Dana Sheahen, Angela Smiley, Amy Bruckman, and Ashok Goel. 2016. Graders as Meta-Reviewers: Simultaneously Scaling and Improving Expert Evaluation for Large Online Classrooms. In Proceedings of the Third (2016) ACM Conference on Learning @ Scale (Edinburgh, Scotland, UK) (L@S ’16). Association for Computing Machinery, New York, NY, USA, 399–408. https://doi.org/10.1145/2876034.2876044

Digital Library

[34]

Deborah S Keller, Emily R Winslow, Joel E Goldberg, and Vanita Ahuja. 2021. Video-based coaching: current status and role in surgical practice (Part 1) from the society for surgery of the alimentary tract, health care quality and outcomes committee. Journal of Gastrointestinal Surgery 25, 9 (2021), 2439–2446.

[35]

Jeongyeon Kim, Daeun Choi, Nicole Lee, Matt Beane, and Juho Kim. 2023. Surch: Enabling Structural Search and Comparison for Surgical Videos. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–17.

Digital Library

[36]

Juho Kim, Elena L Glassman, Andrés Monroy-Hernández, and Meredith Ringel Morris. 2015. RIMES: Embedding interactive multimedia exercises in lecture videos. In Proceedings of the 33rd annual ACM conference on human factors in computing systems. 1535–1544.

Digital Library

[37]

Juho Kim, Philip J Guo, Carrie J Cai, Shang-Wen Li, Krzysztof Z Gajos, and Robert C Miller. 2014. Data-driven interaction techniques for improving navigation of educational videos. In Proceedings of the 27th annual ACM symposium on User interface software and technology. 563–572.

Digital Library

[38]

Juho Kim, Phu Tran Nguyen, Sarah Weir, Philip J. Guo, Robert C. Miller, and Krzysztof Z. Gajos. 2014. Crowdsourcing Step-by-Step Information Extraction to Enhance Existing How-to Videos. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 4017–4026. https://doi.org/10.1145/2556288.2556986

Digital Library

[39]

Juho Kim, Phu Tran Nguyen, Sarah Weir, Philip J Guo, Robert C Miller, and Krzysztof Z Gajos. 2014. Crowdsourcing step-by-step information extraction to enhance existing how-to videos. In Proceedings of the SIGCHI conference on human factors in computing systems. 4017–4026.

Digital Library

[40]

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, 2023. Segment anything. arXiv preprint arXiv:2304.02643 (2023).

[41]

Kadir Kirtac, Nizamettin Aydin, Joël L. Lavanchy, Guido Beldi, Marco Smit, Michael S. Woods, and Florian Aspart. 2022. Surgical Phase Recognition: From Public Datasets to Real-World Data. Applied Sciences 12, 17 (2022). https://doi.org/10.3390/app12178746

[42]

Daichi Kitaguchi, Nobuyoshi Takeshita, Hiroki Matsuzaki, Hiroaki Takano, Yohei Owada, Tsuyoshi Enomoto, Tatsuya Oda, Hirohisa Miura, Takahiro Yamanashi, Masahiko Watanabe, 2020. Real-time automatic surgical phase recognition in laparoscopic sigmoidectomy using the convolutional neural network-based deep learning approach. Surgical endoscopy 34 (2020), 4924–4931.

[43]

Kenneth R Koedinger, Jihee Kim, Julianna Zhuxin Jia, Elizabeth A McLaughlin, and Norman L Bier. 2015. Learning is not a spectator sport: Doing is better than watching for learning from a MOOC. In Proceedings of the second (2015) ACM conference on learning@ scale. 111–120.

Digital Library

[44]

Chinmay E. Kulkarni, Michael S. Bernstein, and Scott R. Klemmer. 2015. PeerStudio: Rapid Peer Feedback Emphasizes Revision and Improves Performance. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale (Vancouver, BC, Canada) (L@S ’15). Association for Computing Machinery, New York, NY, USA, 75–84. https://doi.org/10.1145/2724660.2724670

Digital Library

[45]

Ghader Kurdi, Jared Leo, Bijan Parsia, Uli Sattler, and Salam Al-Emari. 2020. A Systematic Review of Automatic Question Generation for Educational Purposes. International Journal of Artificial Intelligence in Education 30, 1 (01 Mar 2020), 121–204. https://doi.org/10.1007/s40593-019-00186-y

[46]

Andreas Leibetseder and Klaus Schoeffmann. 2020. SurgXplore: Interactive Video Exploration for Endoscopy. In Proceedings of the 2020 International Conference on Multimedia Retrieval (Dublin, Ireland) (ICMR ’20). Association for Computing Machinery, New York, NY, USA, 397–401. https://doi.org/10.1145/3372278.3391930

Digital Library

[47]

Yi Li and Wenzhao Li. 2018. A survey of sketch-based image retrieval. Machine Vision and Applications 29, 7 (2018), 1083–1100.

Digital Library

[48]

Martin Lindvall, Claes Lundström, and Jonas Löwgren. 2021. Rapid Assisted Visual Search: Supporting Digital Pathologists with Imperfect AI. In 26th International Conference on Intelligent User Interfaces (College Station, TX, USA) (IUI ’21). Association for Computing Machinery, New York, NY, USA, 504–513. https://doi.org/10.1145/3397481.3450681

Digital Library

[49]

Ching Liu, Juho Kim, and Hao-Chuan Wang. 2018. ConceptScape: Collaborative concept mapping for video learning. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–12.

Digital Library

[50]

Jakub Lokoč, Gregor Kovalčík, Bernd Münzer, Klaus Schöffmann, Werner Bailer, Ralph Gasser, Stefanos Vrochidis, Phuong Anh Nguyen, Sitapa Rujikietgumjorn, and Kai Uwe Barthel. 2019. Interactive Search or Sequential Browsing? A Detailed Analysis of the Video Browser Showdown 2018. ACM Trans. Multimedia Comput. Commun. Appl. 15, 1, Article 29 (feb 2019), 18 pages. https://doi.org/10.1145/3295663

Digital Library

[51]

Xinyi Lu, Simin Fan, Jessica Houghton, Lu Wang, and Xu Wang. 2023. ReadingQuizMaker: A Human-NLP Collaborative System that Supports Instructors to Design High-Quality Reading Quiz Questions. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–18.

Digital Library

[52]

Lin Ma and Yuchun Ma. 2019. Automatic Question Generation Based on MOOC Video Subtitles and Knowledge Graph. In Proceedings of the 2019 7th International Conference on Information and Education Technology (Aizu-Wakamatsu, Japan) (ICIET 2019). Association for Computing Machinery, New York, NY, USA, 49–53. https://doi.org/10.1145/3323771.3323820

Digital Library

[53]

Mukta Majumder and Sujan Kumar Saha. 2014. Automatic selection of informative sentences: The sentences that can generate multiple choice questions. Knowledge Management & E-Learning: An International Journal 6 (2014), 377–391.

[54]

Mukta Majumder and Sujan Kumar Saha. 2015. A System for Generating Multiple Choice Questions: With a Novel Approach for Sentence Selection. In Proceedings of the 2nd Workshop on Natural Language Processing Techniques for Educational Applications. Association for Computational Linguistics, Beijing, China, 64–72. https://doi.org/10.18653/v1/W15-4410

[55]

Salman Maqbool, Aqsa Riaz, Hasan Sajid, and Osman Hasan. 2020. m2caiSeg: Semantic Segmentation of Laparoscopic Images using Convolutional Neural Networks. arxiv:2008.10134 [cs.CV]

[56]

Justin Matejka, Tovi Grossman, and George Fitzmaurice. 2013. Swifter: improved online video scrubbing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI ’13). Association for Computing Machinery, New York, NY, USA, 1159–1168. https://doi.org/10.1145/2470654.2466149

Digital Library

[57]

Akira Matsuda, Toru Okuzono, Hiromi Nakamura, Hideaki Kuzuoka, and Jun Rekimoto. 2021. A Surgical Scene Replay System for Learning Gastroenterological Endoscopic Surgery Skill by Multiple Synchronized-Video and Gaze Representation. Proc. ACM Hum.-Comput. Interact. 5, EICS, Article 204 (may 2021), 22 pages. https://doi.org/10.1145/3461726

Digital Library

[58]

Laura Mazer, Oliver Varban, John R Montgomery, Michael M Awad, and Allison Schulman. 2022. Video is better: why aren’t we using it? A mixed-methods study of the barriers to routine procedural video recording and case review. Surgical endoscopy (2022), 1–8.

[59]

Helena M. Mentis, Yuanyuan Feng, Azin Semsar, and Todd A. Ponsky. 2020. Remotely Shaping the View in Surgical Telementoring. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376622

Digital Library

[60]

Bill Moggridge and Bill Atkinson. 2007. Designing interactions. Vol. 17. MIT press Cambridge.

[61]

Bernd Muenzer, Klaus Schoeffmann, and Laszlo Böeszöermenyi. 2017. EndoXplore: A Web-Based Video Explorer for Endoscopic Videos. In 2017 IEEE International Symposium on Multimedia (ISM). 366–367. https://doi.org/10.1109/ISM.2017.70

[62]

Megha Nawhal, Jacqueline B. Lang, Greg Mori, and Parmit K. Chilana. 2019. VideoWhiz: Non-Linear Interactive Overviews for Recipe Videos. In Proceedings of Graphics Interface 2019 (Kingston, Ontario) (GI 2019). Canadian Information Processing Society, 8 pages. https://doi.org/10.20380/GI2019.15

Digital Library

[63]

Sravanthi Nittala, Pooja Agarwal, R. Vishnu, and Sahana Shanbhag. 2023. Speaker Diarization and BERT-Based Model for Question Set Generation from Video Lectures. In Information and Communication Technology for Competitive Strategies (ICTCS 2021), Amit Joshi, Mufti Mahmud, and Roshan G. Ragel (Eds.). Springer Nature Singapore, Singapore, 441–452.

[64]

American Board of Surgery. 2019. Operative Performance Rating System. https://www.absurgery.org/default.jsp?certgsqe_resassess

[65]

Amy Pavel, Colorado Reed, Björn Hartmann, and Maneesh Agrawala. 2014. Video Digests: A Browsable, Skimmable Format for Informational Lecture Videos. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (Honolulu, Hawaii, USA) (UIST ’14). Association for Computing Machinery, New York, NY, USA, 573–582. https://doi.org/10.1145/2642918.2647400

Digital Library

[66]

Luca Ponzanelli, Gabriele Bavota, Andrea Mocci, Rocco Oliveto, Massimiliano Di Penta, Sonia Haiduc, Barbara Russo, and Michele Lanza. 2019. Automatic Identification and Classification of Software Development Video Tutorial Fragments. IEEE Transactions on Software Engineering 45, 5 (2019), 464–488. https://doi.org/10.1109/TSE.2017.2779479

Digital Library

[67]

Vitaliy Popov, Xinyue Chen, Jingying Wang, Michael Kemp, Gurjit Sandhu, Taylor Kantor, Natalie Mateju, and Xu Wang. 2024. Looking Together ≠ Seeing the Same Thing: Understanding Surgeons’ Visual Needs During Intra-operative Coordination and Instruction. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, Hawaii, USA) (CHI ’24). Association for Computing Machinery, New York, NY, USA, 17 pages. https://doi.org/10.1145/3613904.3641929

Digital Library

[68]

Manfred Jürgen Primus, Klaus Schoeffmann, and Laszlo Böszörmenyi. 2016. Temporal segmentation of laparoscopic videos into surgical phases. In 2016 14th International Workshop on Content-Based Multimedia Indexing (CBMI). 1–6. https://doi.org/10.1109/CBMI.2016.7500249

[69]

Printio. 2022. Fabric.js. https://github.com/fabricjs/fabric.js.

[70]

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. Springer, 234–241.

[71]

Luca Rossetto, Ivan Giangreco, Heiko Schuldt, Stéphane Dupont, Omar Seddati, Metin Sezgin, and Yusuf Sahillioğlu. 2015. IMOTION—a content-based video retrieval engine. In MultiMedia Modeling: 21st International Conference, MMM 2015, Sydney, NSW, Australia, January 5-7, 2015, Proceedings, Part II 21. Springer, 255–260.

[72]

Mehdi S.M. Sajjadi, Morteza Alamgir, and Ulrike von Luxburg. 2016. Peer Grading in a Course on Algorithms and Data Structures: Machine Learning Algorithms Do Not Improve over Simple Baselines. In Proceedings of the Third (2016) ACM Conference on Learning @ Scale (Edinburgh, Scotland, UK) (L@S ’16). Association for Computing Machinery, New York, NY, USA, 369–378. https://doi.org/10.1145/2876034.2876036

Digital Library

[73]

Sophia M. Schmitz, Sandra Schipper, Martin Lemos, Patrick H. Alizai, Elda Kokott, Jonathan F. Brozat, Ulf P. Neumann, and Tom F. Ulmer. 2021. Development of a tailor‐made surgical online learning platform, ensuring surgical education in times of the COVID19 pandemic. BMC Surgery 21, 1 (17 Apr 2021), 196. https://doi.org/10.1186/s12893-021-01203-5

[74]

Klaus Schoeffmann, Manfred Del Fabro, Tibor Szkaliczki, Laszlo Böszörmenyi, and Jörg Keckstein. 2015. Keyframe extraction in endoscopic video. Multimedia Tools and Applications 74, 24 (01 Dec 2015), 11187–11206. https://doi.org/10.1007/s11042-014-2224-7

Digital Library

[75]

Naomi M. Sell, Douglas J. Cassidy, Sophia K. McKinley, Emil Petrusa, Denise W. Gee, Mara B. Antonoff, and Roy Phitayakorn. 2021. A Needs Assessment of Video-based Education Resources Among General Surgery Residents. Journal of Surgical Research 263 (2021), 116–123. https://doi.org/10.1016/j.jss.2021.01.035

[76]

Azin Semsar, Hannah McGowan, Yuanyuan Feng, H. Reza Zahiri, Adrian Park, Andrea Kleinsmith, and Helena Mentis. 2019. How Trainees Use the Information from Telepointers in Remote Instruction. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 93 (nov 2019), 20 pages. https://doi.org/10.1145/3359195

Digital Library

[77]

Sheng Shen, Yaliang Li, Nan Du, X. Wu, Yusheng Xie, Shen Ge, Tao Yang, Kai Wang, Xin-Fang Liang, and Wei Fan. 2018. On the Generation of Medical Question-Answer Pairs. In AAAI Conference on Artificial Intelligence.

[78]

Hyungyu Shin, Eun-Young Ko, Joseph Jay Williams, and Juho Kim. 2018. Understanding the effect of in-video prompting on learners and instructors. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–12.

Digital Library

[79]

Ken’ichi Shinozuka, Sayaka Turuda, Atsuro Fujinaga, Hiroaki Nakanuma, Masahiro Kawamura, Yusuke Matsunobu, Yuki Tanaka, Toshiya Kamiyama, Kohei Ebe, Yuichi Endo, Tsuyoshi Etoh, Masafumi Inomata, and Tatsushi Tokuyasu. 2022. Artificial intelligence software available for medical devices: surgical phase recognition in laparoscopic cholecystectomy. Surgical Endoscopy 36, 10 (01 Oct 2022), 7444–7452. https://doi.org/10.1007/s00464-022-09160-7

[80]

Bruno Silva, Bruno Oliveira, Pedro Morais, LR Buschle, Jorge Correia-Pinto, Estevão Lima, and Joao L Vilaça. 2022. Analysis of Current Deep Learning Networks for Semantic Segmentation of Anatomical Structures in Laparoscopic Surgery. In 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE, 3502–3505.

[81]

Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).

[82]

Pritam Singh, Rajesh Aggarwal, Muaaz Tahir, Philip H. Pucher, and Ara Darzi. 2015. A Randomized Controlled Study to Evaluate the Role of Video-based Coaching in Training Laparoscopic Skills. Annals of Surgery 261, 5 (2015). https://journals.lww.com/annalsofsurgery/Fulltext/2015/05000/A_Randomized_Controlled_Study_to_Evaluate_the_Role.9.aspx

[83]

Rebecca A. Snyder, Margaret J. Tarpley, John L. Tarpley, Mario Davidson, Colleen Brophy, and Jeffery B. Dattilo. 2012. Teaching in the Operating Room: Results of a National Survey. Journal of Surgical Education 69, 5 (2012), 643–649. https://doi.org/10.1016/j.jsurg.2012.06.007

[84]

Mikael L Soucisse, Kerianne Boulva, Lucas Sideris, Pierre Drolet, Michel Morin, and Pierre Dubé. 2017. Video coaching as an efficient teaching method for surgical residents—a randomized controlled trial. Journal of surgical education 74, 2 (2017), 365–371.

[85]

Mikael L. Soucisse, Kerianne Boulva, Lucas Sideris, Pierre Drolet, Michel Morin, and Pierre Dubé. 2017. Video Coaching as an Efficient Teaching Method for Surgical Residents—A Randomized Controlled Trial. Journal of Surgical Education 74, 2 (2017), 365–371. https://doi.org/10.1016/j.jsurg.2016.09.002

[86]

Anh Truong, Peggy Chi, David Salesin, Irfan Essa, and Maneesh Agrawala. 2021. Automatic Generation of Two-Level Hierarchical Tutorials from Instructional Makeup Videos. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 108, 16 pages. https://doi.org/10.1145/3411764.3445721

Digital Library

[87]

Xu Wang, Simin Fan, Jessica Houghton, and Lu Wang. 2022. Towards Process-Oriented, Modular, and Versatile Question Generation that Meets Educational Needs. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 291–302.

[88]

Xu Wang, Carolyn Rose, and Ken Koedinger. 2021. Seeing Beyond Expert Blind Spots: Online Learning Design for Scale and Quality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 51, 14 pages. https://doi.org/10.1145/3411764.3445045

Digital Library

[89]

Xu Wang, Srinivasa Teja Talluri, Carolyn Rose, and Kenneth Koedinger. 2019. UpGrade: Sourcing Student Open-Ended Solutions to Create Scalable Learning Opportunities. In Proceedings of the Sixth (2019) ACM Conference on Learning @ Scale (Chicago, IL, USA) (L@S ’19). Association for Computing Machinery, New York, NY, USA, Article 17, 10 pages. https://doi.org/10.1145/3330430.3333614

Digital Library

[90]

Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. 2023. Seggpt: Segmenting everything in context. arXiv preprint arXiv:2304.03284 (2023).

[91]

Zichao Wang, Andrew S. Lan, Weili Nie, Andrew E. Waters, Phillip J. Grimaldi, and Richard G. Baraniuk. 2018. QG-Net: A Data-Driven Question Generation Model for Educational Content. In Proceedings of the Fifth Annual ACM Conference on Learning at Scale (London, United Kingdom) (L@S ’18). Association for Computing Machinery, New York, NY, USA, Article 7, 10 pages. https://doi.org/10.1145/3231644.3231654

Digital Library

[92]

Zichao Wang, Jakob Valdez, Debshila Basu Mallick, and Richard G. Baraniuk. 2022. Towards Human-Like Educational Question Generation with Large Language Models. In Artificial Intelligence in Education, Maria Mercedes Rodrigo, Noburu Matsuda, Alexandra I. Cristea, and Vania Dimitrova (Eds.). Springer International Publishing, Cham, 153–166.

[93]

Sarah Weir, Juho Kim, Krzysztof Z Gajos, and Robert C Miller. 2015. Learnersourcing subgoal labels for how-to videos. In Proceedings of the 18th ACM conference on computer supported cooperative work & social computing. 405–416.

Digital Library

[94]

Jiayuan Xie, Yi Cai, Qingbao Huang, and Tao Wang. 2021. Multiple Objects-Aware Visual Question Generation. In Proceedings of the 29th ACM International Conference on Multimedia (Virtual Event, China) (MM ’21). Association for Computing Machinery, New York, NY, USA, 4546–4554. https://doi.org/10.1145/3474085.3476969

Digital Library

[95]

Ying Xu, Valery Vigil, Andres S Bustamante, and Mark Warschauer. 2022. “Elinor’s Talking to Me!”: Integrating Conversational AI into Children’s Narrative Science Programming. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–16.

Digital Library

[96]

Saelyne Yang, Jisu Yim, Juho Kim, and Hijung Valentina Shin. 2022. CatchLive: Real-Time Summarization of Live Streams with Stream Content and Interaction Data. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 500, 20 pages. https://doi.org/10.1145/3491102.3517461

Digital Library

[97]

Iman Yeckehzaare, Tirdad Barghi, and Paul Resnick. 2020. QMaps: Engaging Students in Voluntary Question Generation and Linking. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376882

Digital Library

[98]

Sasi Kiran Yelamarthi, Shiva Krishna Reddy, Ashish Mishra, and Anurag Mittal. 2018. A Zero-Shot Framework for Sketch based Image Retrieval. In Proceedings of the European Conference on Computer Vision (ECCV).

Digital Library

[99]

YouTube. 2013. Laparoscopic cholecystectomy / dangerous artery & harmonic scalpel.https://www.youtube.com/watch?v=YMx3t5RrI6Y

[100]

YouTube. 2016. Critical View of Safety achievement during Laparoscopic Cholecystectomy Op. https://www.youtube.com/watch?v=TGxlc3WECHg&t=278s

[101]

YouTube. 2016. Laparoscopic cholecystectomy (gallbladder surgery). https://www.youtube.com/watch?v=sHoCp169leA

[102]

YouTube. 2018. Laparoscopic Cholecystectomy for Symptomatic Cholelithiasis. https://www.youtube.com/watch?v=HAaVQYBNcMA&t=751s

[103]

YouTube. 2022. Laparoscopic Cholecystectomy: Chronic Calculous Cholecystitis. https://www.youtube.com/watch?v=JD16eOgDO1s&t=3s

[104]

YouTube. 2023. Uncomplicated Laparoscopic Cholecystectomy. https://www.youtube.com/watch?v=O0LcQw2pxkk&t=5s

[105]

Ke Yuan, Dafang He, Zhuoren Jiang, Liangcai Gao, Zhi Tang, and C. Lee Giles. 2019. Automatic Generation of Headlines for Online Math Questions. In AAAI Conference on Artificial Intelligence.

[106]

Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang. 2018. Unet++: A nested u-net architecture for medical image segmentation. In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: 4th International Workshop, DLMIA 2018, and 8th International Workshop, ML-CDS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 20, 2018, Proceedings 4. Springer, 3–11.

Digital Library

[107]

Peide Zhu and Claudia Hauff. 2021. Evaluating BERT-Based Rewards for Question Generation with Reinforcement Learning. In Proceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval (Virtual Event, Canada) (ICTIR ’21). Association for Computing Machinery, New York, NY, USA, 261–270. https://doi.org/10.1145/3471158.3472240

Digital Library

Cited By

Urrea CGarcia-Garcia YKern J(2024)Improving Surgical Scene Semantic Segmentation through a Deep Learning Architecture with Attention to Class ImbalanceBiomedicines10.3390/biomedicines1206130912:6(1309)Online publication date: 13-Jun-2024
https://doi.org/10.3390/biomedicines12061309
Popov VChen XWang JKemp MSandhu GKantor TMateju NWang X(2024)Looking Together ≠ Seeing the Same Thing: Understanding Surgeons' Visual Needs During Intra-operative Coordination and InstructionProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3641929(1-12)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3641929

Index Terms

Surgment: Segmentation-enabled Semantic Search and Creation of Visual Question and Feedback to Support Video-Based Surgery Learning
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interactive systems and tools
2. Social and professional topics
  1. Professional topics
    1. History of computing

Index terms have been assigned to the content through auto-classification.

Recommendations

Surch: Enabling Structural Search and Comparison for Surgical Videos
CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems

Video is an effective medium for learning procedural knowledge, such as surgical techniques. However, learning procedural knowledge through videos remains difficult due to limited access to procedural structures of knowledge (e.g., compositions and ...
Haptic-enabled virtual training in orthognathic surgery
Abstract
Orthognathic surgery (OGS) is a very complex surgical procedure aiming to correct a wide range of skeletal and dental irregularities, including jaws and teeth misalignments. It requires a precise pre-surgical planning and high surgical skills that ...
Students practice minimally invasive surgery through game-based assisted learning
Edutainment'11: Proceedings of the 6th international conference on E-learning and games, edutainment technologies

Minimally invasive surgery (MIS) is revolutionary skill of surgical operation for a surgeon. Minimally invasive surgery which is specialized operationuses miniature cameras with microscopes, tiny fiber-optic flashlights and high definition monitors ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems

May 2024

18961 pages

ISBN:9798400703300

DOI:10.1145/3613904

Editors:
Florian Floyd Mueller
Monash University
,
Penny Kyburz
The Australian National University
,
Julie R. Williamson
University of Glasgow
,
Corina Sas
Lancaster University
,
Max L. Wilson
University of Nottingham
,
Phoebe Toups Dugas
Monash University/New Mexico State University
,
Irina Shklovski
University of Copenhagen

Copyright © 2024 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 May 2024

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Artifacts Available / v1.1

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CHI '24

Sponsor:

CHI '24: CHI Conference on Human Factors in Computing Systems

May 11 - 16, 2024

HI, Honolulu, USA

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI '25

Sponsor:
sigchi

CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
512
Total Downloads

Downloads (Last 12 months)512
Downloads (Last 6 weeks)93

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Urrea CGarcia-Garcia YKern J(2024)Improving Surgical Scene Semantic Segmentation through a Deep Learning Architecture with Attention to Class ImbalanceBiomedicines10.3390/biomedicines1206130912:6(1309)Online publication date: 13-Jun-2024
https://doi.org/10.3390/biomedicines12061309
Popov VChen XWang JKemp MSandhu GKantor TMateju NWang X(2024)Looking Together ≠ Seeing the Same Thing: Understanding Surgeons' Visual Needs During Intra-operative Coordination and InstructionProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3641929(1-12)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3641929

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Table of Contents