Influence of multimodal instructions on learning tool manipulation skills through mentoring in an immersive environment: Influence des instructions multimodales sur l’apprentissage par compagnonnage des compétences de manipulation d’outil dans un environnement immersif
Authors: 
Cassandre Simon, Manel Boukli Hacene, Flavien Lebrun, Samir Otmane, 
Amine ChellaliAuthors Info & Claims
Cassandre Simon, Manel Boukli Hacene, Flavien Lebrun, Samir Otmane, Amine ChellaliAuthors Info & ClaimsArticle No.: 1, Pages 1 - 13
Abstract
La formation par compagnonnage permet aux novices d’acquérir des compétences sous la supervision d’experts qui utilisent diverses modalités de communication. Cependant, reproduire ce modèle dans des simulateurs immersifs reste un défi, notamment pour assurer une communication efficace entre experts et novices. Notre étude explore l’impact de la communication multimodale expert-novice pour transmettre des instructions sur l’amplitude des mouvements dans une tâche de manipulation d’outils en environnement immersif. Les résultats révèlent que la combinaison des modalités visuelle-haptique améliore la précision, la vitesse et la qualité des mouvements. De plus, la combinaison verbale-visuelle-haptique renforce le sentiment de présence et de coprésence, et l’expérience d’apprentissage. Ces résultats suggèrent que la combinaison visuelle-haptique est optimale pour améliorer les performances des novices, et que l’intégration de la modalité verbale améliore l’expérience utilisateur. Ces conclusions ouvrent de nouvelles perspectives pour améliorer l’acquisition de gestes techniques par compagnonnage en réalité virtuelle grâce à la communication multimodale.
Supplemental Material
MP4 File
Presentation video
- Download
- 186.92 MB
References
[1]
Mortaja AlQassab and David Wang. 2016. Human Performance and Cognitive Workload in Multi-sensory Virtual Environments. In Universal Access in Human-Computer Interaction. Interaction Techniques and Environments: 10th International Conference, UAHCI 2016, Held as Part of HCI International 2016, Toronto, ON, Canada, July 17-22, 2016, Proceedings, Part II 10. Springer, 219–230. https://doi.org/10.1007/978-3-319-40244-4_21
[2]
Michael J Barrington, Daniel M Wong, Ben Slater, Jason J Ivanusic, and Matthew Ovens. 2012. Ultrasound-guided regional anesthesia: how much practice do novices require before achieving competency in ultrasound needle visualization using a cadaver model. Regional Anesthesia & Pain Medicine 37, 3 (2012), 334–339. https://doi.org/10.1097/AAP.0b013e3182475fba
[3]
C. Basdogan, C. Ho, M. A. Srinivasan, and M. Slater. 2000. An experimental study on the role of touch in shared virtual environments. ACM Transactions on Computer-Human Interaction (TOCHI) 7 (2000), 443–460. Issue 4. https://doi.org/10.1145/365058.365082
[4]
David Brickler, Matias Volonte, Jeffrey W Bertrand, Andrew T Duchowski, and Sabarish V Babu. 2019. Effects of stereoscopic viewing and haptic feedback, sensory-motor congruence and calibration on near-field fine motor perception-action coordination in virtual reality. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 28–37. https://doi.org/10.1109/VR.2019.8797744
[5]
Dario Cecilio-Fernandes, Fokie Cnossen, Jenifer Coster, AD (Debbie) C Jaarsma, and René A Tio. 2020. The effects of expert and augmented feedback on learning a complex medical skill. Perceptual and motor skills 127, 4 (2020), 766–784. https://doi.org/10.1177/0031512520914680
[6]
Amine Chellali, Woojin Ahn, Ganesh Sankaranarayanan, JT Flinn, Steven D Schwaitzberg, Daniel B Jones, Suvranu De, and Caroline GL Cao. 2015. Preliminary evaluation of the pattern cutting and the ligating loop virtual laparoscopic trainers. Surgical endoscopy 29 (2015), 815–821. https://doi.org/10.1007/s00464-014-3764-7
[7]
Amine Chellali, Cédric Dumas, and Isabelle Milleville-Pennel. 2011. Influences of haptic communication on a shared manual task. Interacting with Computers 23, 4 (2011), 317–328. https://doi.org/10.1016/j.intcom.2011.05.002
[8]
Amine Chellali, Cédric Dumas, and Isabelle Milleville-Pennel. 2012. Haptic communication to support biopsy procedures learning in virtual environments. Presence: Teleoperators and Virtual Environments 21, 4 (2012), 470–489. https://doi.org/10.1162/PRES_a_00128
[9]
Amine Chellali, Isabelle Milleville-Pennel, and Cédric Dumas. 2013. Influence of contextual objects on spatial interactions and viewpoints sharing in virtual environments. Virtual Reality 17, 1 (2013), 1–15. https://doi.org/10.1007/s10055-012-0214-5
[10]
Elizabeth F Churchill and Dave Snowdon. 1998. Collaborative virtual environments: an introductory review of issues and systems. virtual reality 3 (1998), 3–15. https://doi.org/10.1007/BF01409793
[11]
Gregorio Convertino, Helena M Mentis, Aleksandra Slavkovic, Mary Beth Rosson, and John M Carroll. 2011. Supporting common ground and awareness in emergency management planning: A design research project. ACM Transactions on Computer-Human Interaction (TOCHI) 18, 4 (2011), 1–34. https://doi.org/10.1145/2063231.2063236
[12]
Cédric Dabo and Sylvain Connac. 2022. Les représentations des étudiants tuteurs en masso-kinésithérapie sur le dispositif de tutorat entre pairs. Recherches en éducation46 (2022). https://doi.org/10.4000/ree.10150
[13]
Michel Desmurget. 2006. Imitation et apprentissages moteurs: des neurones miroirs à la pédagogie du geste sportif. Groupe de Boeck.
[14]
Oliver Ehmer. 2021. Synchronization in demonstrations. Multimodal practices for instructing body knowledge. Linguistics Vanguard 7, s4 (2021), 20200038. https://doi.org/10.1515/lingvan-2020-0038
[15]
Oliver Ehmer and Geert Brône. 2021. Instructing embodied knowledge: multimodal approaches to interactive practices for knowledge constitution. Linguistics Vanguard 7, s4 (2021), 20210012. https://doi.org/10.1515/lingvan-2021-0012
[16]
Jesse Fox, Sun Joo Ahn, Joris H Janssen, Leo Yeykelis, Kathryn Y Segovia, and Jeremy N Bailenson. 2015. Avatars versus agents: a meta-analysis quantifying the effect of agency on social influence. Human–Computer Interaction 30, 5 (2015), 401–432. https://doi.org/10.1080/07370024.2014.921494
[17]
R Brent Gillespie, M Sile O’Modhrain, Philip Tang, David Zaretzky, and Cuong Pham. 1998. The virtual teacher. In ASME International Mechanical Engineering Congress and Exposition, Vol. 15861. American Society of Mechanical Engineers, 171–178. https://doi.org/10.1115/IMECE1998-0250
[18]
Noah J Glaser and Matthew Schmidt. 2020. Usage considerations of 3D collaborative virtual learning environments to promote development and transfer of knowledge and skills for individuals with autism. Technology, Knowledge and Learning 25, 2 (2020), 315–322. https://doi.org/10.1007/s10758-018-9369-9
[19]
Erving Goffman. 1974. Frame analysis: An essay on the organization of experience.Harvard University Press. https://doi.org/10.2307/2064022
[20]
Carl Gutwin, Oliver Schneider, Robert Xiao, and Stephen Brewster. 2011. Chalk sounds: the effects of dynamic synthesized audio on workspace awareness in distributed groupware. In Proceedings of the ACM 2011 conference on Computer supported cooperative work. 85–94. https://doi.org/10.1145/1958824.1958838
[21]
William S Halsted. 1904. The training of the surgeon. Bull Johns Hop Hosp (1904), 267–275. https://doi.org/10.1001/jama.1904.02500210043006
[22]
Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psychology. Vol. 52. Elsevier, 139–183. https://doi.org/10.1016/s0166-4115(08)62386-9
[23]
John Hattie and Helen Timperley. 2007. The power of feedback. Review of educational research 77, 1 (2007), 81–112. https://doi.org/10.3102/003465430298487
[24]
Morten Hertzum. 2021. Reference values and subscale patterns for the task load index (TLX): a meta-analytic review. Ergonomics 64, 7 (2021), 869–878. https://doi.org/
[25]
Jon Hindmarsh, Mike Fraser, Christian Heath, Steve Benford, and Chris Greenhalgh. 1998. Fragmented interaction: establishing mutual orientation in virtual environments. In Proceedings of the 1998 ACM conference on Computer supported cooperative work. 217–226. https://doi.org/10.1145/289444.289496
[26]
Adrian H Hoppe, Roland Reeb, Florian van de Camp, and Rainer Stiefelhagen. 2018. Interaction of distant and local users in a collaborative virtual environment. In Virtual, Augmented and Mixed Reality: Interaction, Navigation, Visualization, Embodiment, and Simulation: 10th International Conference, VAMR 2018, Held as Part of HCI International 2018, Las Vegas, NV, USA, July 15-20, 2018, Proceedings, Part I 10. Springer, 328–337. https://doi.org/10.1007/978-3-319-91581-4_24
[27]
Keke Lu, Guanyang Liu, and Lingzhi Liu. 2013. A study on haptic collaborative game in shared virtual environment. In Fifth International Conference on Machine Vision (ICMV 2012): Computer Vision, Image Analysis and Processing, Vol. 8783. SPIE, 425–431. https://doi.org/10.1117/12.2021258
[28]
Richard Magill and David I Anderson. 2016. Motor learning and control (11th ed.). McGraw-Hill Publishing New York.
[29]
Daniel Martin, Sandra Malpica, Diego Gutierrez, Belen Masia, and Ana Serrano. 2022. Multimodality in VR: A survey. ACM Computing Surveys (CSUR) 54, 10s (2022), 1–36. https://doi.org/10.1145/3508361
[30]
Helena M Mentis, Amine Chellali, and Steven Schwaitzberg. 2014. Learning to see the body: supporting instructional practices in laparoscopic surgical procedures. In Proceedings of the SIGCHI conference on human factors in computing systems. 2113–2122. https://doi.org/10.1145/2556288.2557387
[31]
M Alex Meredith. 2002. On the neuronal basis for multisensory convergence: a brief overview. Cognitive brain research 14, 1 (2002), 31–40. https://doi.org/10.1016/s0926-6410(02)00059-9
[32]
Jonas Moll, Eva-Lotta Sallnäs Pysander, Kerstin Severinson Eklundh, and Sten-Olof Hellström. 2014. The effects of audio and haptic feedback on collaborative scanning and placing. Interacting with computers 26, 3 (2014), 177–195. https://doi.org/10.1093/iwc/iwt031
[33]
Kristine L. Nowak and Frank Biocca. 2003. The Effect of the Agency and Anthropomorphism of Users’ Sense of Telepresence, Copresence, and Social Presence in Virtual Environments. Presence: Teleoper. Virtual Environ. 12 (2003), 481–494. https://doi.org/10.1162/105474603322761289
[34]
Catherine S Oh, Jeremy N Bailenson, and Gregory F Welch. 2018. A systematic review of social presence: Definition, antecedents, and implications. Frontiers in Robotics and AI 5 (2018), 409295. https://doi.org/10.3389/frobt.2018.00114
[35]
Sofianna Ojala, Joonas Sirola, Timo Nykopp, Heikki Kröger, and Henrik Nuutinen. 2022. The impact of teacher’s presence on learning basic surgical tasks with virtual reality headset among medical students. Medical education online 27, 1 (2022), 2050345. https://doi.org/10.1080/10872981.2022.2050345
[36]
Vasiliki Papageorgiou and Petros Lameras. 2017. Multimodal Teaching and Learning with the Use of Technology: Meanings, Practices and Discourses.International Association for Development of the Information Society (2017).
[37]
Stéphanie Philippe, Alexis D Souchet, Petros Lameras, Panagiotis Petridis, Julien Caporal, Gildas Coldeboeuf, and Hadrien Duzan. 2020. Multimodal teaching, learning and training in virtual reality: a review and case study. Virtual Reality & Intelligent Hardware 2, 5 (2020), 421–442. https://doi.org/10.1016/j.vrih.2020.07.008
[38]
Mark C Porte, George Xeroulis, Richard K Reznick, and Adam Dubrowski. 2007. Verbal feedback from an expert is more effective than self-accessed feedback about motion efficiency in learning new surgical skills. The American journal of surgery 193, 1 (2007), 105–110. https://doi.org/10.1016/j.amjsurg.2006.03.016
[39]
Lauriane Pouliquen-Lardy, Isabelle Milleville-Pennel, Francois Guillaume, and Franck Mars. 2016. Remote collaboration in virtual reality: asymmetrical effects of task distribution on spatial processing and mental workload. Virtual Reality 20 (2016), 213–220. https://doi.org/10.1007/s10055-016-0294-8
[40]
J. Rasmussen. 1983. Skills, rules, knowledge: signals, signs and symbols and other distinctions in human performance models. IEEE Transactions in Systems, Man, and Cybernetics 13 (1983), 257–266. Issue 3. https://doi.org/10.1109/TSMC.1983.6313160
[41]
Aylen Ricca, Amine Chellali, and Samir Otrnane. 2021. The influence of hand visualization in tool-based motor-skills training, a longitudinal study. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR). IEEE, 103–112. https://doi.org/10.1109/VR50410.2021.00031
[42]
Eva-Lotta Sallnäs. 2004. The effect of modality on social presence, presence and performance in collaborative virtual environments. Ph.D. Dissertation. KTH.
[43]
Eva-Lotta Sallnäs, Kirsten Rassmus-Gröhn, and Calle Sjöström. 2000. Supporting presence in collaborative environments by haptic force feedback. ACM Transactions on Computer-Human Interaction (TOCHI) 7, 4 (2000), 461–476. https://doi.org/10.1145/365058.365086
[44]
Richard A Schmidt, Timothy D Lee, Carolee Winstein, Gabriele Wulf, and Howard N Zelaznik. 2018. Motor control and learning: A behavioral emphasis (5th ed.). Human kinetics.
[45]
Roland Sigrist, Georg Rauter, Robert Riener, and Peter Wolf. 2013. Augmented visual, auditory, haptic, and multimodal feedback in motor learning: a review. Psychonomic bulletin & review 20 (2013), 21–53. https://doi.org/10.3758/s13423-012-0333-8
[46]
Cassandre Simon, Manel Boukli Hacene, Samir Otmane, and Amine Chellali. 2023. Study of communication modalities to support teaching tool manipulation skills in a shared immersive environment. Computers and Graphics 117 (2023), 31–41. https://doi.org/10.1016/j.cag.2023.09.011
[47]
Mel Slater, Amela Sadagic, Martin Usoh, and Ralph Schroeder. 2000. Small-group behavior in a virtual and real environment: A comparative study. Presence 9, 1 (2000), 37–51. https://doi.org/10.1162/105474600566600
[48]
Jeanett Strandbygaard, Flemming Bjerrum, Mathilde Maagaard, Per Winkel, Christian Rifbjerg Larsen, Charlotte Ringsted, Christian Gluud, Teodor Grantcharov, Bent Ottesen, and Jette Led Sorensen. 2013. Instructor feedback versus no instructor feedback on performance in a laparoscopic virtual reality simulator: a randomized trial. https://doi.org/10.1097/SLA.0b013e31827eee6e.
[49]
Ankeet D Udani, T Edward Kim, Steven K Howard, and Edward R Mariano. 2015. Simulation in teaching regional anesthesia: current perspectives. Local and regional anesthesia (2015), 33–43. https://doi.org/10.2147/LRA.S68223
[50]
Holly S Vitense, Julie A Jacko, and V Kathlene Emery. 2003. Multimodal feedback: an assessment of performance and mental workload. Ergonomics 46, 1-3 (2003), 68–87. https://doi.org/10.1080/00140130303534
[51]
Mark T Wallace, M Alex Meredith, and Barry E Stein. 1993. Converging influences from visual, auditory, and somatosensory cortices onto output neurons of the superior colliculus. Journal of neurophysiology 69, 6 (1993), 1797–1809. https://doi.org/10.1152/jn.1993.69.6.1797
[52]
Jinling Wang, Amine Chellali, and Caroline GL Cao. 2016. Haptic communication in collaborative virtual environments. Human factors 58, 3 (2016), 496–508. https://doi.org/10.1177/0018720815618808
[53]
Fei Wu, Jerald Thomas, Shreyas Chinnola, and Evan Suma Rosenberg. 2020. Exploring communication modalities to support collaborative guidance in virtual reality. In 2020 IEEE conference on virtual reality and 3d user interfaces abstracts and workshops (VRW). IEEE, 79–86. https://doi.org/10.1109/VRW50115.2020.00021
[54]
Geoff Wyvill, Craig McPheeters, and Brian Wyvill. 1986. Data Structure for soft Objects. The Visual Computer 2, 4 (Aug. 1986), 227–234. https://doi.org/10.1007/BF01900346
[55]
Eugenia Yiannakopoulou, Nikolaos Nikiteas, Despina Perrea, and Christos Tsigris. 2015. Virtual reality simulators and training in laparoscopic surgery. International Journal of Surgery 13 (2015), 60–64. https://doi.org/10.1016/j.ijsu.2014.11.014
Index Terms
- Influence of multimodal instructions on learning tool manipulation skills through mentoring in an immersive environment: Influence des instructions multimodales sur l’apprentissage par compagnonnage des compétences de manipulation d’outil dans un environnement immersif
Recommendations
Effects of Visuospatial Abilities on Learning Through 2D and Immersive Video
IHM '24: Proceedings of the 35th Conference on l'Interaction Humain-MachineLa réalité virtuelle our l’apprentissage de la chirurgie grâce à son immersion offre une approche novatrice prsion. Cependant, son efficacité semble être influencée par l’aptitude visuo-spatiale (AVS) de l’utilisateur. Dans ce travail, nous avons étudié ...
Comments
Information & Contributors
Information
Published In
March 2024
209 pages
ISBN:9798400718113
DOI:10.1145/3649792
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 03 May 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Data Availability
Presentation video https://dl.acm.org/doi/10.1145/3649792.3649793#Video.MP4
Funding Sources
- French National Research Agency (ANR)
Conference
IHM '24
IHM '24: 35th International Francophone Conference on Human-Computer Interaction
March 25 - 29, 2024
Paris, France
Acceptance Rates
Overall Acceptance Rate 103 of 199 submissions, 52%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 13Total Downloads
- Downloads (Last 12 months)13
- Downloads (Last 6 weeks)3
Reflects downloads up to 15 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format