Toward Scalable and Transparent Multimodal Analytics to Study Standard Medical Procedures: Linking Hand Movement, Proximity, and Gaze Data
Pages 3 - 10
Abstract
This study employed multimodal learning analytics (MMLA) to analyze behavioral dynamics during the ABCDE procedure in nursing education, focusing on gaze entropy, hand movement velocities, and proximity measures. Utilizing accelerometers and eye-tracking techniques, behaviorgrams were generated to depict various procedural phases. Results identified four primary phases characterized by distinct patterns of visual attention, hand movements, and proximity to the patient or instruments. The findings suggest that MMLA can offer valuable insights into procedural competence in medical education. This research underscores the potential of MMLA to provide detailed, objective evaluations of clinical procedures and their inherent complexities.
References
[1]
Nima Ahmadi, Farzan Sasangohar, Jing Yang, Denny Yu, Valerie Danesh, Steven Klahn, and Faisal Masud. 2022. Quantifying Workload and Stress in Intensive Care Unit Nurses: Preliminary Evaluation Using Continuous Eye-Tracking. Human factors 0, 0 (May 2022), 187208221085335.
[2]
Haifa Alwahaby, Mutlu Cukurova, Zacharoula Papamitsiou, and Michail Giannakos. 2022. The Evidence of Impact and Ethical Considerations of Multimodal Learning Analytics: A Systematic Literature Review. In The Multimodal Learning Analytics Handbook, Michail Giannakos, Daniel Spikol, Daniele Di Mitri, Kshitij Sharma, Xavier Ochoa, and Rawad Hammad (Eds.). Springer International Publishing, Cham, 289--325.
[3]
Roger Azevedo and Dragan Gašević. 2019. Analyzing Multimodal Multichannel Data about Self-Regulated Learning with Advanced Learning Technologies: Issues and Challenges. Computers in human behavior 96 (July 2019), 207--210.
[4]
Paulo Blikstein. 2013. Multimodal learning analytics. In Proceedings of the Third International Conference on Learning Analytics and Knowledge (Leuven, Belgium) (LAK '13). Association for Computing Machinery, New York, NY, USA, 102--106.
[5]
Birgitta Burger, Anna Puupponen, and Tommi Jantunen. 2018. Synchronizing eye tracking and optical motion capture: How to bring them together. Journal of eye movement research 11, 2 (May 2018), 1--16.
[6]
R M Burian. 1997. Exploratory experimentation and the role of histochemical techniques in the work of Jean Brachet, 1938--1952. History and philosophy of the life sciences 19, 1 (1997), 27--45.
[7]
Jackie S Cha and Denny Yu. 2022. Objective Measures of Surgeon Non-Technical Skills in Surgery: A Scoping Review. Human factors 64, 1 (Feb. 2022), 42--73.
[8]
Richard Chen, Filip Jankovic, Nikki Marinsek, Luca Foschini, Lampros Kourtis, Alessio Signorini, Melissa Pugh, Jie Shen, Roy Yaari, Vera Maljkovic, Marc Sunga, Han Hee Song, Hyun Joon Jung, Belle Tseng, and Andrew Trister. 2019. Developing Measures of Cognitive Impairment in the Real World from Consumer-Grade Multimodal Sensor Streams. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (Anchorage, AK, USA) (KDD '19). Association for Computing Machinery, New York, NY, USA, 2145--2155.
[9]
Elizabeth B Cloude, Megan D Wiedbusch, Daryn A Dever, Dario Torre, and Roger Azevedo. 2022. The Role of Metacognition and Self-regulation on Clinical Reasoning: Leveraging Multimodal Learning Analytics to Transform Medical Education. In The Multimodal Learning Analytics Handbook, Michail Giannakos, Daniel Spikol, Daniele Di Mitri, Kshitij Sharma, Xavier Ochoa, and Rawad Hammad (Eds.). Springer International Publishing, Cham, 105--129.
[10]
G Conole, D Gašević, P Long, and G Siemens. 2011. Message from the LAK 2011 General & Program Chairs. In Proceedings of the 1st International Conference on Learning Analytics and Knowledge. ACM, New York, NY, 1--2.
[11]
Wenqiang Cui. 2019. Visual Analytics: A Comprehensive Overview. IEEE Access 7 (2019), 81555--81573.
[12]
Lida Z David, Maaike D Endedijk, and Piet Van den Bossche. 2022. Investigating Interaction Dynamics: A Temporal Approach to Team Learning. In Methods for Researching Professional Learning and Development: Challenges, Applications and Empirical Illustrations, Michael Goller, Eva Kyndt, Susanna Paloniemi, and Crina Damşa (Eds.). Springer International Publishing, Cham, 187--209.
[13]
Berna Devezer and Erkan Buzbas. 2021. Minimum Viable Experiment to Replicate. http://philsci-archive.pitt.edu/21475/
[14]
Daniele Di Mitri, Jan Schneider, Marcus Specht, and Hendrik Drachsler. 2018. From signals to knowledge: A conceptual model for multimodal learning analytics. Journal of Computer Assisted Learning 34, 4 (2018), 338--349.
[15]
Farina Faiz, Yoshinori Ideno, Hiromichi Iwasaki, Yoko Muroi, and Sozo Inoue. 2021. Multilabel Classification of Nursing Activities in a Realistic Scenario. In Activity and Behavior Computing, Md Atiqur Rahman Ahad, Sozo Inoue, Daniel Roggen, and Kaori Fujinami (Eds.). Springer Singapore, Singapore, 269--288.
[16]
Felipe Fernández-Méndez, Martín Otero-Agra, Cristian Abelairas-Gómez, Nieves María Sáez-Gallego, Antonio Rodriguez-Núñez, and Roberto Barcala-Furelos. 2019. ABCDE approach to victims by lifeguards: How do they manage a critical patient? A cross sectional simulation study. PloS one 14, 4 (April 2019), e0212080.
[17]
Gloria Fernandez-Nieto, Roberto Martinez-Maldonado, Vanessa Echeverria, Kirsty Kitto, Pengcheng An, and Simon Buckingham Shum. 2021. What Can Analytics for Teamwork Proxemics Reveal About Positioning Dynamics In Clinical Simulations? Proc. ACM Hum.-Comput. Interact. 5, CSCW1 (April 2021), 1--24.
[18]
Michael C Frank, Edward Vul, and Scott P Johnson. 2009. Development of infants' attention to faces during the first year. Cognition 110, 2 (Feb. 2009), 160--170.
[19]
Gerd Gigerenzer. 1991. From tools to theories: A heuristic of discovery in cognitive psychology. Psychological review 98, 2 (April 1991), 254--267.
[20]
Zhenyu Gu, Chenhao Jin, Danny Chang, and Liqun Zhang. 2021. Predicting webpage aesthetics with heatmap entropy. Behaviour & information technology 40, 7 (May 2021), 676--690.
[21]
Ville Heilala, Päivikki Jääskelä, Mirka Saarela, Anna-Stina Kuula, Anne Eskola, and Tommi Kärkkäinen. 2022. "Sitting at the Stern and Holding the Rudder": Teachers' Reflections on Action in Higher Education Based on Student Agency Analytics. In Digital Teaching and Learning in Higher Education: Developing and Disseminating Skills for Blended Learning, Leonid Chechurin (Ed.). Palgrave Macmillan, Cham, 71--91.
[22]
Kenneth Holmqvist, Marcus Nyström, Richard Andersson, Richard Dewhurst, Halszka Jarodzka, and Joost van de Weijer. 2011. Eye Tracking: A comprehensive guide to methods and measures. OUP Oxford, Oxford.
[23]
Edwin Hutchins. 1995. Cognition in the Wild. MIT Press, Cambridge, MA.
[24]
Malek Karaim, Aboelmagd Noureldin, and Tashfeen B Karamat. 2019. Low-cost IMU Data Denoising using Savitzky-Golay Filters. In 2019 International Conference on Communications, Signal Processing, and their Applications (ICCSPA). IEEE, Piscataway, 1--5.
[25]
Daniel A Keim. 2001. Visual exploration of large data sets. Commun. ACM 44, 8 (Aug. 2001), 38--44.
[26]
Daniel A Keim. 2002. Information visualization and visual data mining. IEEE transactions on visualization and computer graphics 8, 1 (Jan. 2002), 1--8.
[27]
Dae-Yeob Kim, Soo-Hyung Kim, Daeseon Choi, and Seung-Hun Jin. 2015. Accurate Indoor Proximity Zone Detection Based on Time Window and Frequency with Bluetooth Low Energy. Procedia computer science 56 (Jan. 2015), 88--95.
[28]
Michaela Kolbe and Margarete Boos. 2019. Laborious but Elaborate: The Benefits of Really Studying Team Dynamics. Frontiers in psychology 10 (June 2019), 1478.
[29]
Joni Lämsä, Joonas Mannonen, Ari Tuhkala, Ville Heilala, Arto Helovuo, Ilkka Tynkkynen, Emilia Lampi, Katriina Sipiläinen, Tommi Kärkkäinen, and Raija Hämäläinen. 2023. Capturing cognitive load management during authentic virtual reality flight training with behavioural and physiological indicators. Journal of computer assisted learning 39, 5 (April 2023), 1553--1563.
[30]
Sara Lanini-Maggi, Ian T Ruginski, Thomas F Shipley, Christophe Hurter, Andrew T Duchowski, Benny B Briesemeister, Jihyun Lee, and Sara I Fabrikant. 2021. Assessing how visual search entropy and engagement predict performance in a multiple-objects tracking air traffic control task. Computers in Human Behavior Reports 4 (Aug. 2021), 100127.
[31]
Joy Yeonjoo Lee, Jeroen Donkers, Halszka Jarodzka, and Jeroen J G van Merriënboer. 2019. How prior knowledge affects problem-solving performance in a medical simulation game: Using game-logs and eye-tracking. Computers in human behavior 99 (Oct. 2019), 268--277.
[32]
Maximilian Xiling Li, Mario Nadj, Alexander Maedche, Dirk Ifenthaler, and Johannes Wöhler. 2021. Towards a physiological computing infrastructure for researching students' flow in remote learning. Technology Knowledge and Learning 27, 1 (Sept. 2021), 365--384.
[33]
Joakim Linja, Joonas Hämäläinen, Paavo Nieminen, and Tommi Kärkkäinen. 2023. Feature selection for distance-based regression: An umbrella review and a one-shot wrapper. Neurocomputing 518 (2023), 344--359.
[34]
David J C MacKay. 2003. Information Theory, Inference and Learning Algorithms. Cambridge University Press, Cambridge.
[35]
Roberto Martinez-Maldonado, Vanessa Echeverria, Gloria Fernandez-Nieto, Lixiang Yan, Linxuan Zhao, Riordan Alfredo, Xinyu Li, Samantha Dix, Hollie Jaggard, Rosie Wotherspoon, Abra Osborne, Simon Buckingham Shum, and Dragan Gašević. 2023. Lessons Learnt from a Multimodal Learning Analytics Deployment In-the-wild. ACM Trans. Comput.-Hum. Interact. 31, 1 (Sept. 2023), 1--41.
[36]
Roberto Martinez-Maldonado, Jurgen Schulte, Vanessa Echeverria, Yuveena Gopalan, and Simon Buckingham Shum. 2020. Where is the teacher? Digital analytics for classroom proxemics. Journal of Computer Assisted Learning 16 (May 2020), 1.
[37]
Inge Molenaar, Susanne de Mooij, Roger Azevedo, Maria Bannert, Sanna Järvelä, and Dragan Gašević. 2023. Measuring self-regulated learning and the role of AI: Five years of research using multimodal multichannel data. Computers in human behavior 139 (Feb. 2023), 107540.
[38]
Kaveh Momen and Geoff R Fernie. 2010. Nursing activity recognition using an inexpensive game controller: An application to infection control. Technology and health care: official journal of the European Society for Engineering and Medicine 18, 6 (2010), 393--408.
[39]
Akio Morita, Yasuo Murai, and Mamoru Mitsuishi. 2022. Current and Future Microsurgical Skills Assessment. In Learning and Career Development in Neurosurgery: Values-Based Medical Education, Ahmed Ammar (Ed.). Springer International Publishing, Cham, 349--356.
[40]
Hamza Ouhaichi, Daniel Spikol, and Bahtijar Vogel. 2023. Research trends in multimodal learning analytics: A systematic mapping study. Computers and Education: Artificial Intelligence 4 (Jan. 2023), 100136.
[41]
Ronald W Schafer. 2011. What Is a Savitzky-Golay Filter? [Lecture Notes]. IEEE Signal Processing Magazine 28, 4 (July 2011), 111--117.
[42]
Nino H C Schoeber, Marjolein Linders, Mathijs Binkhorst, Willem-Pieter De Boode, Jos M T Draaisma, Marlies Morsink, Anneliese Nusmeier, Martijn Pas, Christine van Riessen, Nigel M Turner, Rutger Verhage, Cornelia R M G Fluit, and Marije Hogeveen. 2022. Healthcare professionals' knowledge of the systematic ABCDE approach: a cross-sectional study. BMC emergency medicine 22, 1 (Dec. 2022), 202.
[43]
C E Shannon. 1948. A mathematical theory of communication. The Bell system technical journal 27, 3 (1948), 379--423.
[44]
Brook Shiferaw, Luke Downey, and David Crewther. 2019. A review of gaze entropy as a measure of visual scanning efficiency. Neuroscience and biobehavioral reviews 96 (Jan. 2019), 353--366.
[45]
Friedrich Steinle. 1997. Entering New Fields: Exploratory Uses of Experimentation. Philosophy of science 64, S4 (1997), S65--S74.
[46]
Troels Thim, Niels Henrik Vinther Krarup, Erik Lerkevang Grove, Claus Valter Rohde, and Bo Løfgren. 2012. Initial assessment and treatment with the Airway, Breathing, Circulation, Disability, Exposure (ABCDE) approach. International journal of general medicine 5 (Jan. 2012), 117--121.
[47]
Junko Tokuno, Tamara E Carver, and Gerald M Fried. 2023. Measurement and Management of Cognitive Load in Surgical Education: A Narrative Review. Journal of surgical education 80, 2 (Feb. 2023), 208--215.
[48]
Philip H Winne. 2019. Paradigmatic Dimensions of Instrumentation and Analytic Methods in Research on Self-Regulated Learning. Computers in human behavior 96 (July 2019), 285--289.
[49]
Marcelo Worsley, Roberto Martinez-Maldonado, and Cynthia D'Angelo. 2021. A New Era in Multimodal Learning Analytics: Twelve Core Commitments to Ground and Grow MMLA. Journal of Learning Analytics 8, 3 (2021), 10--27.
[50]
Andrew G Wright, Rahool Patel, Koraly Perez-Edgar, Xiaoxue Fu, Kayla Brown, Sanjib Adhikary, and Adrian Zurca. 2022. Eye-Tracking Technology to Determine Procedural Proficiency in Ultrasound-Guided Regional Anesthesia. The journal of education in perioperative medicine : JEPM 24, 1 (Jan. 2022), E684.
[51]
Elaheh Yadegaridehkordi, Nurul Fazmidar Binti Mohd Noor, Mohamad Nizam Bin Ayub, Hannyzzura Binti Affal, and Nornazlita Binti Hussin. 2019. Affective computing in education: A systematic review and future research. Computers & education 142 (Dec. 2019), 103649.
[52]
Lixiang Yan, Linxuan Zhao, Dragan Gasevic, and Roberto Martinez-Maldonado. 2022. Scalability, Sustainability, and Ethicality of Multimodal Learning Analytics. In LAK22: 12th International Learning Analytics and Knowledge Conference (Online, USA) (LAK22). Association for Computing Machinery, New York, NY, USA, 13--23.
[53]
Linxuan Zhao, Zachari Swiecki, Dragan Gasevic, Lixiang Yan, Samantha Dix, Hollie Jaggard, Rosie Wotherspoon, Abra Osborne, Xinyu Li, Riordan Alfredo, and Roberto Martinez-Maldonado. 2023. METS: Multimodal Learning Analytics of Embodied Teamwork Learning. In LAK23: 13th International Learning Analytics and Knowledge Conference (Arlington, TX, USA) (LAK2023). Association for Computing Machinery, New York, NY, USA, 186--196.
Index Terms
- Toward Scalable and Transparent Multimodal Analytics to Study Standard Medical Procedures: Linking Hand Movement, Proximity, and Gaze Data
Recommendations
Gaze-Hand Alignment: Combining Eye Gaze and Mid-Air Pointing for Interacting with Menus in Augmented Reality
ETRAGaze and freehand gestures suit Augmented Reality as users can interact with objects at a distance without need for a separate input device. We propose Gaze-Hand Alignment as a novel multimodal selection principle, defined by concurrent use of both gaze ...
Hand motion gestural oscillations and multimodal discourse
ICMI '03: Proceedings of the 5th international conference on Multimodal interfacesTo develop multimodal interfaces, one needs to understand the constraints underlying human communicative gesticulation and the kinds of features one may compute based on these underlying human characteristics.In this paper we address hand motion ...
Assisted Music Score Reading Using Fixed-Gaze Head Movement: Empirical Experiment and Design Implications
Eye-tracking has a very strong potential in human computer interaction (HCI) as an input modality, particularly in mobile situations. However, it lacks convenient action triggering methods. In our research, we investigate the combination of eye-tracking ...
Comments
Information & Contributors
Information
Published In
April 2024
1898 pages
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 21 May 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Research Council of Finland
Conference
SAC '24
Sponsor:
Acceptance Rates
Overall Acceptance Rate 1,650 of 6,669 submissions, 25%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 183Total Downloads
- Downloads (Last 12 months)183
- Downloads (Last 6 weeks)122
Reflects downloads up to 16 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in