survey

Automatic Performance Assessment in Three-dimensional Interactive Haptic Medical Simulators: A Systematic Review

Authors:

Lucas H. Sallaberry,

Fátima L. S. NunesAuthors Info & Claims

ACM Computing Surveys, Volume 55, Issue 7

Article No.: 131, Pages 1 - 35

https://doi.org/10.1145/3539222

Published: 15 December 2022 Publication History

Abstract

This study presents a literature systematic review of automatic performance assessment in three-dimensional interactive medical and dental simulators with haptic feedback, resulting in 63 included articles. The main contributions regard analysis and discussion of investigated procedures, extracted metrics, experiment types, and assessment techniques. Studies have mostly focused on assessing performance by analyzing metrics using statistical techniques, while machine learning algorithms appear to be underexplored. Metrics related to time were observed in 84% of the studies, and aspects related to force, error and precision were investigated to a lesser degree. Difficulties reported in the articles are discussed, and research opportunities are presented.

References

[1]

IEEE. 2021. IEEE Xplore. Retrieved 12 January 2021 from https://ieeexplore.ieee.org/Xplore/home.jsp.

[2]

National Library of Medicine. 2021. PubMed. Retrieved 12 January 2021 from https://pubmed.ncbi.nlm.nih.gov/.

[3]

Elsevier. 2021. Scopus. Retrieved 30 December 2020 from https://www.elsevier.com/solutions/scopus.

[4]

Association for Computing Machinery. 2021. ACM Digital Library. Retrieved 12 January 2021 from https://dl.acm.org/.

[5]

Kashif Akhtar, Kapil Sugand, Matthew Sperrin, Justin Cobb, Nigel Standfield, and Chinmay Gupte. 2015. Training safer orthopedic surgeons: Construct validation of a virtual-reality simulator for hip fracture surgery. Acta Orthopaed. 86, 5 (Sept.2015), 616–621. DOI:

[6]

Abdullatif Aydin, Nicholas Raison, Muhammad Shamim Khan, Prokar Dasgupta, and Kamran Ahmed. 2016. Simulation-based training and assessment in urological surgery. Nat. Rev. Urol. 13, 9 (Sept.2016), 503–519. DOI:

[7]

Hamed Azarnoush, Samaneh Siar, Robin Sawaya, Gmaan Al Zhrani, Alexander Winkler-Schwartz, Fahad Eid Alotaibi, Abdulgadir Bugdadi, Khalid Bajunaid, Ibrahim Marwa, Abdulrahman Jafar Sabbagh, and Rolando F. Del Maestro. 2017. The force pyramid: A spatial analysis of force application during virtual reality brain tumor resection. J. Neurosurg. 127, 1 (July2017), 171–181. DOI:

[8]

P. Pat Banerjee, Cristian J. Luciano, G. Michael Lemole, Fady T. Charbel, and Michael Y. Oh. 2007. Accuracy of ventriculostomy catheter placement using a head- and hand-tracked high-resolution virtual reality simulator with haptic feedback. J. Neurosurg. 107, 3 (Sept.2007), 515–521. DOI:

[9]

James G. Bittner, John D. Mellinger, Toufic Imam, Robert R. Schade, and Bruce V. MacFadyen. 2010. Face and construct validity of a computer-based virtual reality simulator for ERCP. Gastrointest. Endosc. 71, 2 (Feb.2010), 357–364. DOI:

[10]

S. M. B. I. Botden, I. H. J. T. de Hingh, and J. J. Jakimowicz. 2009. Suturing training in augmented reality: Gaining proficiency in suturing skills faster. Surgic. Endosc. 23, 9 (Sept.2009), 2131–2137. DOI:

[11]

Samy Bouaicha, Susanne Epprecht, Thorsten Jentzsch, Lukas Ernstbrunner, Rany El Nashar, and Stefan Rahm. 2020. Three days of training with a low-fidelity arthroscopy triangulation simulator box improves task performance in a virtual reality high-fidelity virtual knee arthroscopy simulator. Knee Surg., Sports Traumatol., Arthros. 28, 3 (Mar.2020), 862–868. DOI:

[12]

James Brewin, Tim Nedas, Ben Challacombe, Oussama Elhage, Jonas Keisu, and Prokar Dasgupta. 2010. Face, content and construct validation of the first virtual reality laparoscopic nephrectomy simulator. BJU Int. 106, 6 (Sept.2010), 850–854. DOI:

[13]

Abdulgadir Bugdadi, Robin Sawaya, Khalid Bajunaid, Duaa Olwi, Alexander Winkler-Schwartz, Nicole Ledwos, Ibrahim Marwa, Ghusn Alsideiri, Abdulrahman Jafar Sabbagh, Fahad E. Alotaibi, Gmaan Al-Zhrani, and Rolando Del Maestro. 2019. Is virtual reality surgical performance influenced by force feedback device utilized? J. Surgic. Educ. 76, 1 (Jan.2019), 262–273. DOI:

[14]

A. Chellali, W. Ahn, G. Sankaranarayanan, J. T. Flinn, S. D. Schwaitzberg, D. B. Jones, Suvranu De, and C. G. L. Cao. 2015. Preliminary evaluation of the pattern cutting and the ligating loop virtual laparoscopic trainers. Surgic. Endosc. 29, 4 (Apr.2015), 815–821. DOI:

[15]

Hong-En Chen, Mary A. Yovanoff, David F. Pepley, Cheyenne C. Sonntag, Katelin A. Mirkin, David C. Han, Jason Z. Moore, and Scarlett R. Miller. 2019. Can haptic simulators distinguish expert performance? A case study in Central Venous Catheterization in Surgical Education:. Simul. Healthc.: J. Societ. Simul. Healthc. 14, 1 (Feb.2019), 35–42. DOI:

[16]

Timothy R. Coles, Dwight Meglan, and Nigel W. John. 2011. The role of haptics in medical training simulators: A survey of the state of the art. IEEE Trans. Haptics 4, 1 (Jan.2011), 51–66. DOI:

Digital Library

[17]

Elen Collaço, Elisabeti Kira, Lucas H. Sallaberry, Anna C. M. Queiroz, Maria A. A. M. Machado, Oswaldo Crivello Jr, and Romero Tori. 2020. Immersion and haptic feedback impacts on dental anesthesia technical skills virtual reality training. J. Dent. Educ. (Nov.2020), jdd.12503. DOI:

[18]

David A. Cook, Ryan Brydges, Benjamin Zendejas, Stanley J. Hamstra, and Rose Hatala. 2013. Technology-enhanced simulation to assess health professionals: A systematic review of validity evidence, research methods, and reporting quality. Acad. Med. 88, 6 (June2013), 872–883. DOI:

[19]

Cléber Gimenez Corrêa, Maria Aparecida de Andrade Moreira Machado, Edith Ranzini, Romero Tori, and Fátima L. S. Nunes. 2017. Virtual reality simulator for dental anesthesia training in the inferior alveolar nerve block. J. Appl. Oral Sci. 25, 4 (Aug.2017), 357–366. DOI:

[20]

Cléber G. Corrêa, Fátima L. S. Nunes, Edith Ranzini, Ricardo Nakamura, and Romero Tori. 2019. Haptic interaction for needle insertion training in medical applications: The state-of-the-art. Med. Eng. Phys. 63 (Jan.2019), 6–25. DOI:

[21]

Suvranu De, Dhannanjay Deo, Ganesh Sankaranarayanan, and Venkata S. Arikatla. 2011. A physics-driven neural networks-based simulation system (PhyNNeSS) for multimodal interactive virtual environments involving nonlinear deformable objects. Pres.: Teleop. Virt. Environ. 20, 4 (Aug.2011), 289–308. DOI:

Digital Library

[22]

Mateus De Lara Ribeiro, Fátima L. S. Nunes, and Simone Elias. 2016. Towards determining force feedback parameters for realistic representation of nodules in a breast palpation simulator. In IEEE 29th International Symposium on Computer-based Medical Systems (CBMS). IEEE, 279–284. DOI:

[23]

Dhanannjay Deo and Suvranu De. 2010. A higher order polynomial reproducing radial basis function neural network (HOPR-RBFN) for real-time interactive simulations of nonlinear deformable bodies with haptic feedback. In IEEE Haptics Symposium. IEEE, 527–530. DOI:

Digital Library

[24]

Roger D. Dias, Avni Gupta, and Steven J. Yule. 2019. Using machine learning to assess physician competence: A systematic review. Acad. Med. 94, 3 (Mar.2019), 427–439. DOI:

[25]

Vivian E. Ekkelenkamp, Arjun D. Koch, Robert A. de Man, and Ernst J. Kuipers. 2016. Training and competence assessment in GI endoscopy: A systematic review. Gut 65, 4 (Apr.2016), 607–615. DOI:

[26]

Alexander Engelhardt, Rajesh Kanawade, Christian Knipfer, Matthias Schmid, Florian Stelzle, and Werner Adler. 2014. Comparing classification methods for diffuse reflectance spectra to improve tissue specific laser surgery. BMC Med. Res. Methodol. 14, 1 (Dec.2014), 91. DOI:

[27]

David Escobar-Castillejos, Julieta Noguez, Fernando Bello, Luis Neri, Alejandra J. Magana, and Bedrich Benes. 2020. A review of training and guidance systems in medical surgery. Appl. Sci. 10, 17 (Aug.2020), 5752. DOI:

[28]

David Escobar-Castillejos, Julieta Noguez, Luis Neri, Alejandra Magana, and Bedrich Benes. 2016. A review of simulators with haptic devices for medical training. J. Med. Syst. 40, 4 (Apr.2016), 104. DOI:

Digital Library

[29]

H. Esen, A. Sachsenhauser, K. Yano, and M. Buss. 2007. A multi-user virtual training system concept and objective assessment of trainings. In 16th IEEE International Symposium on Robot and Human Interactive Communication. IEEE1084–1089. DOI:

[30]

Elizabeth J. Eve, Samuel Koo, Abdulmonem A. Alshihri, Jeremy Cormier, Maria Kozhenikov, R. Bruce Donoff, and Nadeem Y. Karimbux. 2014. Performance of dental students versus prosthodontics residents on a 3D immersive haptic simulator. J. Dent. Educ. 78, 4 (Apr.2014), 630–637. DOI:

[31]

Te-Yung Fang, Pa-Chun Wang, Chih-Hsien Liu, Mu-Chun Su, and Shih-Ching Yeh. 2014. Evaluation of a haptics-based virtual reality temporal bone simulator for anatomy and surgery training. Comput. Meth. Prog. Biomed. 113, 2 (Feb.2014), 674–681. DOI:

Digital Library

[32]

Andrew Feifer, Josee Delisle, and Maurice Anidjar. 2008. Hybrid augmented reality simulator: Preliminary construct validation of laparoscopic smoothness in a urology residency program. J. Urol. 180, 4 (Oct.2008), 1455–1459. DOI:

[33]

Gaelle Fiard, Sonia-Yuki Selmi, Emmanuel Promayon, Lucile Vadcard, Jean-Luc Descotes, and Jocelyne Troccaz. 2014. Initial validation of a virtual-reality learning environment for prostate biopsies: Realism matters! J. Endourol. 28, 4 (Apr.2014), 453–458. DOI:

[34]

Sandro F. Fucentese, Stefan Rahm, Karl Wieser, Jonas Spillmann, Matthias Harders, and Peter P. Koch. 2015. Evaluation of a virtual-reality-based simulator using passive haptic feedback for knee arthroscopy. Knee Surg., Sports Traumatol., Arthros. 23, 4 (Apr.2015), 1077–1085. DOI:

[35]

Nicholas Gélinas-Phaneuf, Nusrat Choudhury, Ahmed R. Al-Habib, Anne Cabral, Etienne Nadeau, Vincent Mora, Valerie Pazos, Patricia Debergue, Robert DiRaddo, and Rolando F. Del Maestro. 2014. Assessing performance in brain tumor resection using a novel virtual reality simulator. Int. J. Comput. Assist. Radiol. Surg. 9, 1 (Jan.2014), 1–9. DOI:

[36]

Christopher J. Gordon, Tayne Ryall, and Belinda Judd. 2016. Simulation-based assessments in health professional education: A systematic review. J. Multidisc. Healthc. (Feb.2016), 69. DOI:

[37]

M. Graafland, J. M. Schraagen, and M. P. Schijven. 2012. Systematic review of serious games for medical education and surgical skills training. Brit. J. Surg. 99, 10 (Sept.2012), 1322–1330. DOI:

[38]

Zhaoxiang Guo, Yonghang Tai, Zhibao Qin, Xiaoqiao Huang, Qiong Li, Jun Peng, and Junsheng Shi. 2020. Development and assessment of a haptic-enabled holographic surgical simulator for renal biopsy training. Soft Comput. 24, 8 (Apr.2020), 5783–5794. DOI:

[39]

Kristine Hagelsteen, Richard Johansson, Mikael Ekelund, Anders Bergenfelz, and Magnus Anderberg. 2019. Performance and perception of haptic feedback in a laparoscopic 3D virtual reality simulator. Minim. Invas. Therap. Allied Technol. 28, 5 (Sept.2019), 309–316. DOI:

[40]

A. Haycock, P. Bassett, J. Bladen, and S. Thomas-Gibson. 2009. Validation of the second-generation olympus colonoscopy simulator for skills assessment. Endoscopy 41, 11 (Oct.2009), 952–958. DOI:

[41]

Caiwen Huang, Horace Cheng, Yves Bureau, Hanif M. Ladak, and Sumit K. Agrawal. 2018. Automated metrics in a virtual-reality myringotomy simulator: Development and construct validity. Otol. Neurotol. 39, 7 (Aug.2018), e601–e608. DOI:

[42]

Jason Jerald. 2016. The VR Book: Human-centered Design for Virtual Reality (1st ed.).Association for Computing Machinery, New York.

[43]

Jing Ren, R. V. Patel, K. A. McIsaac, G. Guiraudon, and T. M. Peters. 2008. Dynamic 3-D virtual fixtures for minimally invasive beating heart procedures. IEEE Trans. Med. Imag. 27, 8 (Aug.2008), 1061–1070. DOI:

[44]

Ernur Karadogan and Robert L. Williams. 2012. Haptic modules for training in palpatory diagnosis. In IEEE Haptics Symposium (HAPTICS). IEEE, 223–230. DOI:

[45]

Toma Kato, Kazuyoshi Tagawa, Takafumi Marutani, Hiromi Tanaka, Masaru Komori, Yoshimasa Kurumi, and Shigehiro Morikawa. 2014. Evaluation of haptic teaching approaches for laparoscopic surgery training. Stud. Health Technol. Inform. 196 (2014), 192–196.

[46]

Sergei Kurenov, Juan Cendan, Saleh Dindar, Kristopher Attwood, James Hassett, Ruth Nawotniak, Gregory Cherr, William G. Cance, and Jörg Peters. 2017. Surgeon-authored virtual laparoscopic adrenalectomy module is judged effective and preferred over traditional teaching tools. Surgic. Innov. 24, 1 (Feb.2017), 72–81. DOI:

[47]

Francis Lau and Joanna Bates. 2004. A review of e-learning practices for undergraduate medical education. J. Med. Syst. 28, 1 (Feb.2004), 71–87. DOI:

Digital Library

[48]

Justin LeBlanc, Carol Hutchison, Yaoping Hu, and Tyrone Donnon. 2013. A comparison of orthopaedic resident performance on surgical fixation of an ulnar fracture using virtual reality and synthetic models. J. Bone Joint Surg. 95, 9 (May2013), e60. DOI:

[49]

Heather Lesch, Evan Johnson, Jörg Peters, and Juan C. Cendán. 2020. VR simulation leads to enhanced procedural confidence for surgical trainees. J. Surgic. Educ. 77, 1 (Jan.2020), 213–218. DOI:

[50]

Min Li, Yun-Hui Liu, and Qiang Huang. 2007. An optimized haptic interaction model based on support vector regression for evaluation of endodontic shaping skill. In IEEE International Conference on Robotics and Biomimetics (ROBIO). IEEE, 617–622. DOI:

[51]

Yanping Lin, Xudong Wang, Fule Wu, Xiaojun Chen, Chengtao Wang, and Guofang Shen. 2014. Development and validation of a surgical training simulator with haptic feedback for learning bone-sawing skill. J. Biomed. Inform. 48 (Apr.2014), 122–129. DOI:

Digital Library

[52]

X. Ma, P. N. Brett, M. T. Wright, and M. V. Griffiths. 2004. A flexible digit with tactile feedback for invasive clinical applications. Proc. Instit. Mech. Eng., Part H: J. Eng. Med. 218, 3 (Mar.2004), 151–158. DOI:

[53]

Avril D. McCarthy, Louise Moody, Alan R. Waterworth, and Derek R. Bickerstaff. 2006. Passive haptics in a knee arthroscopy simulator: Is it valid for core skills training? Clin. Orthopaed. Relat. Res. 442, &NA; (Jan.2006), 13–20. DOI:

[54]

Elspeth M. McDougall. 2007. Validation of surgical simulators. J. Endourol. 21, 3 (Mar.2007), 244–247. DOI:

[55]

I. Mirghani, F. Mushtaq, M. J. Allsop, L. M. Al-Saud, N. Tickhill, C. Potter, A. Keeling, M. A. Mon-Williams, and M. Manogue. 2018. Capturing differences in dental training using a virtual reality simulator. Eur. J. Dent. Educ. 22, 1 (Feb.2018), 67–71. DOI:

[56]

M. B. Molinero, G. Dagnino, J. Liu, W. Chi, M. E. M. K. Abdelaziz, T. M. Y. Kwok, C. Riga, and G. Z. Yang. 2019. Haptic guidance for robot-assisted endovascular procedures: Implementation and evaluation on surgical simulator. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 5398–5403. DOI:

Digital Library

[57]

Morgan Passiment, Heather Sacks, and Grace Huang. 2011. Medical Simulation in Medical Education: Results of an AAMC Survey. Association of American Medical Colleges.

[58]

Usue Mori, Alexander Mendiburu, and José Antonio Lozano. 2016. Distance measures for time series in R: The tsdist package. R Journal 8, 2 (2016), 451.

[59]

D. Morris, C. Sewell, F. Barbagli, K. Salisbury, N. H. Blevins, and S. Girod. 2006. Visuohaptic simulation of bone surgery for training and evaluation. IEEE Comput. Graph. Applic. 26, 6 (Nov.2006), 48–57. DOI:

Digital Library

[60]

Mohammad F. Obeid, Robert E. Kelly, and Frederic D. McKenzie. 2021. Development and validation of a hybrid nuss procedure surgical simulator and trainer. IEEE Trans. Biomed. Eng. (2021), 1–1. DOI:

[61]

Robert V. O’Toole, Robert R. Playter, Thomas M. Krummel, William C. Blank, Nancy H. Cornelius, Webb R. Roberts, Whitney J. Bell, and Marc Raibert. 1999. Measuring and developing suturing technique with a virtual reality surgical simulator. J. Amer. Coll. Surg. 189, 1 (July1999), 114–127. DOI:

[62]

Aswathi P, Balu M. Menon, and Bhavani R. Rao. 2018. Performance categorization for personalized learning in vocational training simulators. In IEEE 18th International Conference on Advanced Learning Technologies (ICALT). IEEE, 66–68. DOI:

[63]

I. G. Papanikolaou, D. Haidopoulos, M. Paschopoulos, I. Chatzipapas, D. Loutradis, and N. F. Vlahos. 2019. Changing the way we train surgeons in the 21st century: A narrative comparative review focused on box trainers and virtual reality simulators. Eur. J. Obstet. Gynecol. Reprod. Biol. 235 (Apr.2019), 13–18. DOI:

[64]

Zachary Pezzementi, Daniel Ursu, Sarthak Misra, and Allison M. Okamura. 2008. Modeling realistic tool-tissue interactions with haptic feedback: A learning-based method. In Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. IEEE, 209–215. DOI:

Digital Library

[65]

M. S. Raghu Prasad, Muniyandi Manivannan, Govindan Manoharan, and S. M. Chandramohan. 2016. Objective assessment of laparoscopic force and psychomotor skills in a novel virtual reality-based haptic simulator. J. Surgic. Educ. 73, 5 (Sept.2016), 858–869. DOI:

[66]

Raghu Prasad, Manivannan Muniyandi, Govindan Manoharan, and Servarayan. M. Chandramohan. 2018. Face and construct validity of a novel virtual reality–based bimanual laparoscopic force-skills trainer with haptics feedback. Surgic. Innov. 25, 5 (Oct.2018), 499–514. DOI:

[67]

Malek Racy, Alastair Barrow, James Tomlinson, and Fernando Bello. 2020. Development and validation of a virtual reality haptic femoral nailing simulator. J. Surgic. Educ. (Nov.2020), S1931720420303810. DOI:

[68]

Hedyeh Rafii-Tari, Christopher J. Payne, Colin Bicknell, Ka-Wai Kwok, Nicholas J. W. Cheshire, Celia Riga, and Guang-Zhong Yang. 2017. Objective assessment of endovascular navigation skills with force sensing. Ann. Biomed. Eng. 45, 5 (May2017), 1315–1327. DOI:

[69]

Stefan Rahm, Marco Germann, Andreas Hingsammer, Karl Wieser, and Christian Gerber. 2016. Validation of a virtual reality-based simulator for shoulder arthroscopy. Knee Surg., Sports Traumatol., Arthros. 24, 5 (May2016), 1730–1737. DOI:

[70]

Karan Rangarajan, Heather Davis, and Philip H. Pucher. 2020. Systematic review of virtual haptics in surgical simulation: A valid educational tool? J. Surgic. Educ. 77, 2 (Mar.2020), 337–347. DOI:

[71]

Carol E. Reiley, Henry C. Lin, David D. Yuh, and Gregory D. Hager. 2011. Review of methods for objective surgical skill evaluation. Surgic. Endosc. 25, 2 (Feb.2011), 356–366. DOI:

[72]

Rob Reilink, Stefano Stramigioli, Astrid M. L. Kappers, and Sarthak Misra. 2011. Evaluation of flexible endoscope steering using haptic guidance. Int. J. Med. Robot. Comput. Assist. Surg. 7, 2 (June2011), 178–186. DOI:

[73]

P. Rhienmora, P. Haddawy, P. Khanal, S. Suebnukarn, and M. N. Dailey. 2010. A virtual reality simulator for teaching and evaluating dental procedures. Meth. Inf. Med. 49, 04 (2010), 396–405. DOI:

[74]

Phattanapon Rhienmora, Peter Haddawy, Siriwan Suebnukarn, and Matthew N. Dailey. 2011. Intelligent dental training simulator with objective skill assessment and feedback. Artif. Intell. Med. 52, 2 (June2011), 115–121. DOI:

[75]

Sama Ria, Margaret J. Cox, Barry F. Quinn, Jonathan P. San Diego, Ali Bakir, and Mark J. Woolford. 2018. A scoring system for assessing learning progression of dental students’ clinical skills using haptic virtual workstations. J. Dent. Educ. 82, 3 (Mar.2018), 277–285. DOI:

[76]

Ben Z. Roitberg, Patrick Kania, Cristian Luciano, Naga Dharmavaram, and Pat Banerjee. 2015. Evaluation of sensory and motor skills in neurosurgery applicants using a virtual reality neurosurgical simulator: The sensory-motor quotient. J. Surgic. Educ. 72, 6 (Nov.2015), 1165–1171. DOI:

[77]

J. Rosen, J. D. Brown, L. Chang, M. N. Sinanan, and B. Hannaford. 2006. Generalized approach for modeling minimally invasive surgery as a stochastic process using a discrete Markov model. IEEE Trans. Biomed. Eng. 53, 3 (Mar.2006), 399–413. DOI:

[78]

J. Rosen, B. Hannaford, C. G. Richards, and M. N. Sinanan. 2001. Markov modeling of minimally invasive surgery based on tool/tissue interaction and force/torque signatures for evaluating surgical skills. IEEE Trans. Biomed. Eng. 48, 5 (May2001), 579–591. DOI:

[79]

Jacob Rosen, Christina Richards, Blake Hannaford, and Mika Sinanan. 2000. Hidden Markov models of minimally invasive surgery. Stud. Health Technol. Inform.-Med. Meets Virt. Real. 70 (2000), 279–285.

[80]

Soroush Sadeghnejad, Farshad Khadivar, Ehsan Abdollahi, Hamed Moradi, Farzam Farahmand, Seyed Mousa Sadr Hosseini, and Gholamreza Vossoughi. 2019. A validation study of a virtual-based haptic system for endoscopic sinus surgery training. Int. J. Med. Robot. Comput. Assist. Surg. 15, 6 (Dec.2019). DOI:

[81]

Juan A. Sánchez-Margallo, Francisco M. Sánchez-Margallo, Ignacio Oropesa, and Enrique J. Gómez. 2014. Systems and technologies for objective evaluation of technical skills in laparoscopic surgery. Minim. Invas. Ther. Allied Technol. 23, 1 (Jan.2014), 40–51. DOI:

[82]

Ganesh Sankaranarayanan, James D. Adair, Tansel Halic, Mark A. Gromski, Zhonghua Lu, Woojin Ahn, Daniel B. Jones, and Suvranu De. 2011. Validation of a novel laparoscopic adjustable gastric band simulator. Surgic. Endosc. 25, 4 (Apr.2011), 1012–1018. DOI:

[83]

Mona Meral Savran, Stine Maya Dreier Sørensen, Lars Konge, Martin G. Tolsgaard, and Flemming Bjerrum. 2016. Training and assessment of hysteroscopic skills: A systematic review. J. Surgic. Educ. 73, 5 (Sept.2016), 906–918. DOI:

[84]

Martina Scafà, Eleonora Brandoni Serrani, Alessandra Papetti, Agnese Brunzini, and Michele Germani. 2020. Assessment of students’ cognitive conditions in medical simulation training: A review study. In Advances in Human Factors and Simulation, Daniel N. Cassenti (Ed.). Vol. 958. Springer International Publishing, Cham, 224–233. DOI:

[85]

Richard A. Schmidt and Timothy Donald Lee. 2014. Motor Learning and Performance: From Principles to Application (5th ed.). Human Kinetics, Champaign, IL. BF295.S248 2014

[86]

Connie C. Schmitz, Debra DaRosa, Maura E. Sullivan, Shari Meyerson, Ken Yoshida, and James R. Korndorffer. 2014. Development and verification of a taxonomy of assessment metrics for surgical technical skills:. Acad. Med. 89, 1 (Jan.2014), 153–161. DOI:

[87]

Sophia F. Shakur, Cristian J. Luciano, Patrick Kania, Ben Z. Roitberg, P. Pat Banerjee, Konstantin V. Slavin, Jeffrey Sorenson, Fady T. Charbel, and Ali Alaraj. 2015. Usefulness of a virtual reality percutaneous trigeminal rhizotomy simulator in neurosurgical training. Operat. Neurosurg. 11, 3 (Sept.2015), 420–425. DOI:

[88]

Shameema Sikder, Jia Luo, P. Pat Banerjee, Cristian Luciano, Patrick Kania, Jonathan Song, Eman Saeed Kahtani, Deepak Edward, and Abdul-Elah Al Towerki. 2015. The use of a virtual reality surgical simulator for cataract surgical skill assessment with 6 months of intervening operating room experience. Clin. Ophthalmol. (Jan.2015), 141. DOI:

[89]

Samaneh Siyar, Hamed Azarnoush, Saeid Rashidi, and Rolando F. Del Maestro. 2020. Tremor assessment during virtual reality brain tumor resection. J. Surgic. Educ. 77, 3 (May2020), 643–651. DOI:

[90]

Pavel Smutny, Marek Babiuch, and Petr Foltynek. 2019. A review of the virtual reality applications in education and training. In 20th International Carpathian Control Conference (ICCC). IEEE, 1–4. DOI:

[91]

P. Ström, A. Kjellin, L. Hedman, E. Johnson, T. Wredmark, and L. Felländer-Tsai. 2003. Validation and learning in the procedicus KSA virtual reality surgical simulator. Surgic. Endosc. 17, 2 (Feb.2003), 227–231. DOI:

[92]

Myat Su Yin, Peter Haddawy, Siriwan Suebnukarn, Holger Schultheis, and Phattanapon Rhienmora. 2017. Use of haptic feedback to train correct application of force in endodontic surgery. In 22nd International Conference on Intelligent User Interfaces. ACM, 451–455. DOI:

Digital Library

[93]

Siriwan Suebnukarn, Peter Haddawy, Phattanapon Rhienmora, Pannapa Jittimanee, and Piyanuch Viratket. 2010. Augmented kinematic feedback from haptic virtual reality for dental skill acquisition. J. Dent. Educ. 74, 12 (Dec.2010), 1357–1366. DOI:

[94]

Siriwan Suebnukarn, Nattharat Phatthanasathiankul, Sunantha Sombatweroje, Phattanapon Rhienmora, and Peter Haddawy. 2009. Process and outcome measures of expert/novice performance on a haptic virtual reality system. J. Dent. 37, 9 (Sept.2009), 658–665. DOI:

[95]

Yonghang Tai, Lei Wei, Hailing Zhou, Saeid Nahavandi, Junsheng Shi, Qiong Li, and Feiyan Li. 2017. A novel framework for visuo-haptic percutaneous therapy simulation based on patient-specific clinical trials. In IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 3362–3366. DOI:

Digital Library

[96]

Yonghang Tai, Lei Wei, Hailing Zhou, Jun Peng, Qiong Li, Feiyan Li, Jun Zhang, and Junsheng Shi. 2019. Augmented-reality-driven medical simulation platform for percutaneous nephrolithotomy with cybersecurity awareness. Int. J. Distrib. Sensor Netw. 15, 4 (Apr.2019), 155014771984017. DOI:

[97]

Yonghang Tai, Lei Wei, Hailing Zhou, Jun Peng, Junsheng Shi, Qiong Li, and Saeid Nahavandi. 2018. Development of haptic-enabled virtual reality simulator for video-assisted thoracoscopic right upper lobectomy. In IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 3010–3015. DOI:

Digital Library

[98]

Anthony S. Thijssen and Marlies P. Schijven. 2010. Contemporary virtual reality laparoscopy simulators: Quicksand or solid grounds for assessing surgical trainees? Amer. J. Surg. 199, 4 (Apr.2010), 529–541. DOI:

[99]

Geb W. Thomas, Brian D. Johns, J. Lawrence Marsh, and Donald D. Anderson. 2014. A review of the role of simulation in developing and assessing orthopaedic surgical skills. Iowa Orthopaed. J. 34 (2014), 181–189.

[100]

Ann Sofia Skou Thomsen. 2017. Intraocular surgery—Assessment and transfer of skills using a virtual-reality simulator. Acta Ophthalm. 95 (June2017), 1–22. DOI:

[101]

Ann Sofia S. Thomsen, Yousif Subhi, Jens Folke Kiilgaard, Morten la Cour, and Lars Konge. 2015. Update on simulation-based surgical training and assessment in ophthalmology. Ophthalmology 122, 6 (June2015), 1111–1130.e1. DOI:

[102]

Costas S. Tzafestas, Kostas Birbas, Yiannis Koumpouros, and Dimitris Christopoulos. 2008. Pilot evaluation study of a virtual paracentesis simulator for skill training and assessment: The beneficial effect of haptic display. Pres.: Teleop. Virt. Environ. 17, 2 (Apr.2008), 212–229. DOI:

Digital Library

[103]

O. A. J. van der Meijden and M. P. Schijven. 2009. The value of haptic feedback in conventional and robot-assisted minimal invasive surgery and virtual reality training: A current review. Surgic. Endosc. 23, 6 (June2009), 1180–1190. DOI:

[104]

P. D. van Hove, G. J. M. Tuijthof, E. G. G. Verdaasdonk, L. P. S. Stassen, and J. Dankelman. 2010. Objective assessment of technical surgical skills. Brit. J. Surg. 97, 7 (May2010), 972–987. DOI:

[105]

Cecilie Våpenstad, Erlend Fagertun Hofstad, Lars Eirik Bø, Magdalena Karolina Chmarra, Esther Kuhry, Gjermund Johnsen, Ronald Mårvik, and Thomas Langø. 2013. Limitations of haptic feedback devices on construct validity of the LapSim® virtual reality simulator. Surgic. Endosc. 27, 4 (Apr.2013), 1386–1396. DOI:

[106]

Cecilie Våpenstad, Erlend Fagertun Hofstad, Thomas Langø, Ronald Mårvik, and Magdalena Karolina Chmarra. 2013. Perceiving haptic feedback in virtual reality simulators. Surgic. Endosc. 27, 7 (July2013), 2391–2397. DOI:

[107]

Neil Vaughan and Venketesh N. Dubey. 2016. Hip replacement simulator for predicting dislocation risk. In 36th Computers and Information in Engineering Conference. American Society of Mechanical Engineers, V01BT02A044. DOI:

[108]

Neil Vaughan and Bogdan Gabrys. 2020. Scoring and assessment in medical VR training simulators with dynamic time series classification. Eng. Applic. Artif. Intell. 94 (Sept.2020), 103760. DOI:

[109]

Dangxiao Wang, Yuru Zhang, Jianxia Hou, Yong Wang, Peijun Lv, Yonggang Chen, and Hui Zhao. 2012. iDental: A haptic-based dental simulator and its preliminary user evaluation. IEEE Trans. Haptics 5, 4 (2012), 332–343. DOI:

Digital Library

[110]

Dangxiao Wang, Siming Zhao, Teng Li, Yuru Zhang, and Xiaoyan Wang. 2015. Preliminary evaluation of a virtual reality dental simulation system on drilling operation. Bio-Med. Mater. Eng. 26, s1 (Aug.2015), S747–S756. DOI:

[111]

Yu Wang, Fan Yang, Yaxin Li, Tao Yang, Cheng Ren, and Zhihang Shi. 2020. A tactile sensation assisted VR catheterization training system for operator’s cognitive skills enhancement. IEEE Access 8 (2020), 57180–57191. DOI:

[112]

Annie Weber, Christopher Domes, Matthew Christian, Max Coale, Cullen Griffith, Nathan N. O’Hara, R. Frank Henn, Robert V. O’Toole, and Marcus F. Sciadini. 2019. Effect of training modules on hip fracture surgical skills simulation performance: Early validation of the AAOS/OTA simulator. J. Bone Joint Surg. 101, 22 (Nov.2019), 2051–2060. DOI:

[113]

Wing-Yin Chan, Jing Qin, Yim-Pan Chui, and Pheng-Ann Heng. 2012. A serious game for learning ultrasound-guided needle placement skills. IEEE Trans. Inf. Technol. Biomed. 16, 6 (Nov.2012), 1032–1042. DOI:

Digital Library

[114]

Alexander Winkler-Schwartz, Vincent Bissonnette, Nykan Mirchi, Nirros Ponnudurai, Recai Yilmaz, Nicole Ledwos, Samaneh Siyar, Hamed Azarnoush, Bekir Karlik, and Rolando F. Del Maestro. 2019. Artificial intelligence in medical education: Best practices using machine learning to assess surgical expertise in virtual reality simulation. J. Surgic. Educ. 76, 6 (Nov.2019), 1681–1690. DOI:

[115]

Xiao Xiao, Shang Zhao, Yan Meng, Lamia Soghier, Xiaoke Zhang, and James Hahn. 2020. A physics-based virtual reality simulation framework for neonatal endotracheal intubation. In IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 557–565. DOI:

[116]

Ehsan Zahedi, Fariba Khosravian, Weiqi Wang, Mehran Armand, Javad Dargahi, and Mehrdad Zadeh. 2020. Towards skill transfer via learning-based guidance in human-robot interaction: An application to orthopaedic surgical drilling skill. J. Intell. Robot. Syst. 98, 3–4 (June2020), 667–678. DOI:

[117]

M. Zhou, S. Tse, A. Derevianko, D. B. Jones, S. D. Schwaitzberg, and C. G. L. Cao. 2012. Effect of haptic feedback in laparoscopic surgery skill acquisition. Surgic. Endosc. 26, 4 (Apr.2012), 1128–1134. DOI:

Cited By

Frata Furlan Peres FNunes FTeixeira JMaurício CConceição KYoshida L(2024)Methods for Evaluating Immersive 3D Virtual Environments: a Systematic Literature ReviewProceedings of the 26th Symposium on Virtual and Augmented Reality10.1145/3691573.3691595(140-151)Online publication date: 30-Sep-2024
https://dl.acm.org/doi/10.1145/3691573.3691595

Index Terms

Automatic Performance Assessment in Three-dimensional Interactive Haptic Medical Simulators: A Systematic Review

Recommendations

Automatic performance assessment in Virtual Reality medical simulators: A model based on procedure trajectories and machine learning
Abstract
Virtual Reality (VR) simulators for medical training helps students develop the sensorimotor skills required to perform specific procedures. However, as a training resource, a VR simulator usually lacks the capacity to assess the user’s ...
Highlights
- Two labeling methods: Participant’s level of experience and instructor’s assessment.
- Better results were obtained using datasets with the instructor’s assessment label.
- Out of five classifiers, the Support Vector Machine achieved ...
Evaluation of haptic virtual reality user interfaces for medical marking on 3D models
Highlights
- Two haptic VR interfaces are proposed to interact with volumetric medical images for improving current computer-aided medical diagnosis and planning.
- Compared with the traditional 2D interface using a screen-based 2D display and a ...
Abstract
Three-dimensional (3D) visualization has been widely used in computer-aided medical diagnosis and planning. To interact with 3D models, current user interfaces in medical systems mainly rely on the traditional 2D interaction techniques by ...
Effects of Haptic Feedback on User Perception and Performance in Interactive Projected Augmented Reality
IXR '22: Proceedings of the 1st Workshop on Interactive eXtended Reality

By means of vibrotactile and force feedback, i.e., haptics, users are given the sensation of touching and manipulating virtual objects in interactive Extended Reality (XR) environments. However, research towards the influence of this feedback on the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Computing Surveys

ACM Computing Surveys Volume 55, Issue 7

July 2023

813 pages

ISSN:0360-0300

EISSN:1557-7341

DOI:10.1145/3567472

Editor:
Albert Zomaya
University of Sydney, Australia

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 December 2022

Online AM: 08 June 2022

Accepted: 06 May 2022

Revised: 17 February 2022

Received: 27 May 2021

Published in CSUR Volume 55, Issue 7

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Survey
Refereed

Funding Sources

Brazilian Federal Agency for Post-Graduation Education (CAPES)
Brazilian National Council of Scientific and Technological Development
São Paulo Research Foundation (FAPESP)
National Institute of Science and Technology - Medicine Assisted by Scientific Computing (INCT-MACC)

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
511
Total Downloads

Downloads (Last 12 months)125
Downloads (Last 6 weeks)19

Reflects downloads up to 14 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Frata Furlan Peres FNunes FTeixeira JMaurício CConceição KYoshida L(2024)Methods for Evaluating Immersive 3D Virtual Environments: a Systematic Literature ReviewProceedings of the 26th Symposium on Virtual and Augmented Reality10.1145/3691573.3691595(140-151)Online publication date: 30-Sep-2024
https://dl.acm.org/doi/10.1145/3691573.3691595

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents