Work in Progress

AR in the OR: exploring use of augmented reality to support endoscopic surgery

Authors:

Edward G. Andrews,

Paul A. Gardner,

Arka N. Mallela,

Jeffrey R. Head,

Joseph C. Maroon,

Georgios A. Zenonos,

Dmitriy Babichenko,

Jacob T. BiehlAuthors Info & Claims

IMX '22: Proceedings of the 2022 ACM International Conference on Interactive Media Experiences

Pages 267 - 270

https://doi.org/10.1145/3505284.3532970

Published: 22 June 2022 Publication History

Abstract

Modern operating rooms (OR) are equipped with several ceiling- and wall-mounted screens that display surgical information. These physical displays are restricted in placement, limiting the surgeons’ ability to freely position them in the environment. Our work addresses this issue by exploring the feasibility of using an augmented reality (AR) headset (Microsoft HoloLens 2) as an alternative to traditional surgical screens; leading to a reduced OR footprint and improved surgical ergonomics. We developed several prototypes using state-of-the-art hardware/software and conducted various neurosurgery-related exploratory studies. Initial feedback from users suggests that coloration and resolution of the holographic feed were adequate, however, surgeons frequently commented on tactile/visual asynchrony. This emphasizes the need for novel, more efficient hardware/software solutions to support fine motor tasks in the OR.

References

[1]

2018. Magic Leap. https://www.magicleap.com/en-us

[2]

2019. Elgato HD60 S+. https://www.elgato.com/en/game-capture-hd60-s-plus

[3]

2019. Holographic Remoting Player. https://docs.microsoft.com/en-us/windows/mixed-reality/develop/native/holographic-remoting-player

[4]

2019. Microsoft HoloLens 2. https://www.microsoft.com/en-us/hololens/hardware

[5]

2019. Microsoft MixedReality-WebRTC. https://github.com/microsoft/MixedReality-WebRTC

[6]

2020. Microsoft Dynamics 365 Remote Assist. https://dynamics.microsoft.com/en-us/mixed-reality/remote-assist/

[7]

2022. Apple AR. https://www.apple.com/augmented-reality/

[8]

2022. CranialMap 3.0 – Stryker Neurosurgial and Advanced Guidance Technologies. https://neurosurgical.stryker.com/products/cranialmap-3/

[9]

2022. Google AR & VR. https://arvr.google.com/ar/

[10]

2022. KARL STORZ Endoscopy. https://www.karlstorz.com/

[11]

2022. Minimally Invasive Neuroendoscopy. https://www.hopkinsmedicine.org/brain-tumor/treatment/surgery/neuroendoscopy.html/

[12]

Jung Kweon Bae, Andrey Vavilin, Joon S You, Hyeongeun Kim, Seon Young Ryu, Jeong Hun Jang, and Woonggyu Jung. 2017. Smartphone-based endoscope system for advanced point-of-care diagnostics: feasibility study. JMIR mHealth and uHealth 5, 7 (2017), e7232.

[13]

Athanasios Christopoulos, Nikolaos Pellas, Justyna Kurczaba, and Robert Macredie. 2021. The effects of augmented reality-supported instruction in tertiary-level medical education. British Journal of Educational Technology(2021).

[14]

Jacob T Gibby, Samuel A Swenson, Steve Cvetko, Raj Rao, and Ramin Javan. 2019. Head-mounted display augmented reality to guide pedicle screw placement utilizing computed tomography. International journal of computer assisted radiology and surgery 14, 3(2019), 525–535.

[15]

Ye Li, Xiaolei Chen, Ning Wang, Wenyao Zhang, Dawei Li, Lei Zhang, Xin Qu, Weitao Cheng, Yueqiao Xu, Wenjin Chen, 2018. A wearable mixed-reality holographic computer for guiding external ventricular drain insertion at the bedside. Journal of neurosurgery 131, 5 (2018), 1599–1606.

[16]

Max Schneider, Christian Kunz, Andrej Pal’a, Christian Rainer Wirtz, Franziska Mathis-Ullrich, and Michal Hlaváč. 2021. Augmented reality–assisted ventriculostomy. Neurosurgical focus 50, 1 (2021), E16.

[17]

Charles Spence and Sarah Squire. 2003. Multisensory integration: maintaining the perception of synchrony. Current Biology 13, 13 (2003), R519–R521.

[18]

Yoshimori Sugano, Mirjam Keetels, and Jean Vroomen. 2010. Adaptation to motor-visual and motor-auditory temporal lags transfer across modalities. Experimental brain research 201, 3 (2010), 393–399.

[19]

Frederick Van Gestel, Taylor Frantz, Cédric Vannerom, Anouk Verhellen, Anthony G Gallagher, Shirley A Elprama, An Jacobs, Ronald Buyl, Michaël Bruneau, Bart Jansen, 2021. The effect of augmented reality on the accuracy and learning curve of external ventricular drain placement. Neurosurgical focus 51, 2 (2021), E8.

[20]

Erik J. Van Lindert, J. André Grotenhuis, and Tjemme Beems. 2004. The use of a head-mounted display for visualization in neuroendoscopy. Computer Aided Surgery 9, 6 (2004), 251–256.

[21]

Jang W Yoon, Robert E Chen, Karim ReFaey, Roberto J Diaz, Ronald Reimer, Ricardo J Komotar, Alfredo Quinones-Hinojosa, Benjamin L Brown, and Robert E Wharen. 2017. Technical feasibility and safety of image-guided parieto-occipital ventricular catheter placement with the assistance of a wearable head-up display. The International Journal of Medical Robotics and Computer Assisted Surgery 13, 4(2017), e1836.

Cited By

Khan TLindlbauer D(2024)Don’t Block My Stuff: Fostering Personal Object Awareness in Multi-user Mixed Reality EnvironmentsProceedings of the ACM on Human-Computer Interaction10.1145/36981268:ISS(20-43)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3698126
Tashiro YMiyafuji SKojima YKiyofuji SKin TIgarashi TKoike H(2024)MR Microsurgical Suture Training System with Level-Appropriate SupportProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642324(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642324
Hurt GKhan TKann MAndrews EBiehl J(2024)EndovasculAR: Utility of Mixed Reality to Segment Large Displays in Surgical Settings2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW62533.2024.00326(1059-1060)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VRW62533.2024.00326
Show More Cited By

Recommendations

Augmented Reality Support for Oncological Neurosurgery
SVR '24: Proceedings of the 26th Symposium on Virtual and Augmented Reality

This project aims to enhance oncological neurosurgery by displaying brain tumors using Augmented Reality (AR) as an overlay image during surgical procedures. Its purpose is to assist neurosurgeons in tasks such as brain biopsies by providing a clear and ...
A HoloLens Based Augmented Reality Navigation System for Minimally Invasive Total Knee Arthroplasty
Intelligent Robotics and Applications
Abstract
In minimally invasive total knee arthroplasty (MIS-TKA), it is difficult to obtain a global vision of the surgical site in the course of surgery without using X-ray imaging, which brings radiation hazard. To provide the surgeon with accurate image ...
Augmented Reality in Orthognathic Surgery: A Multi-Modality Tracking Approach to Assess the Temporomandibular Joint Motion
Extended Reality
Abstract
Augmented Reality (AR) is a rapidly emerging technology finding growing applications in various surgery domains. In this study, we develop and test the feasibility of a novel AR application for Microsoft HoloLens2 Head Mounted Display (HMD) to ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

IMX '22: Proceedings of the 2022 ACM International Conference on Interactive Media Experiences

June 2022

390 pages

ISBN:9781450392129

DOI:10.1145/3505284

Copyright © 2022 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 June 2022

Check for updates

Author Tags

Qualifiers

Work in progress
Research
Refereed limited

Funding Sources

Tull Foundation

Conference

IMX '22

Sponsor:

IMX '22: ACM International Conference on Interactive Media Experiences

June 22 - 24, 2022

JB, Aveiro, Portugal

Acceptance Rates

Overall Acceptance Rate 69 of 245 submissions, 28%

Upcoming Conference

IMX '25

Sponsor:
sigchi

ACM International Conference on Interactive Media Experiences

June 3 - 6, 2025

Niter?i , Brazil

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
229
Total Downloads

Downloads (Last 12 months)54
Downloads (Last 6 weeks)9

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Khan TLindlbauer D(2024)Don’t Block My Stuff: Fostering Personal Object Awareness in Multi-user Mixed Reality EnvironmentsProceedings of the ACM on Human-Computer Interaction10.1145/36981268:ISS(20-43)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3698126
Tashiro YMiyafuji SKojima YKiyofuji SKin TIgarashi TKoike H(2024)MR Microsurgical Suture Training System with Level-Appropriate SupportProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642324(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642324
Hurt GKhan TKann MAndrews EBiehl J(2024)EndovasculAR: Utility of Mixed Reality to Segment Large Displays in Surgical Settings2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW62533.2024.00326(1059-1060)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VRW62533.2024.00326
Khan TAndrews EBiehl J(2024)Future of Surgical Mixed Reality: Cutting-Edge or Cutting Too Deep?2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW62533.2024.00041(203-207)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VRW62533.2024.00041
Ibrahim MMajumder AGopi MSayadi LVyas R(2024)Illuminating precise stencils on surgical sites using projection-based augmented realitySmart Health10.1016/j.smhl.2024.10047632(100476)Online publication date: Jun-2024
https://doi.org/10.1016/j.smhl.2024.100476
Millard KRainouard F(2023)Convergence of mathematical and physical optical designs for the development of random photonic integrated circuits for an unconventional AR display conceptOptical Architectures for Displays and Sensing in Augmented, Virtual, and Mixed Reality (AR, VR, MR) IV10.1117/12.2669115(142)Online publication date: 16-Mar-2023
https://doi.org/10.1117/12.2669115
Millard KRainouard FFowler DGhibaudo EMartinez C(2023)Multilayer photonic-integrated circuit design for dense random waveguide distribution addressing, application to near-eye displayOptical Architectures for Displays and Sensing in Augmented, Virtual, and Mixed Reality (AR, VR, MR) IV10.1117/12.2649858(137)Online publication date: 16-Mar-2023
https://doi.org/10.1117/12.2649858
Da Silva DCassimiro DDantas CMaciel P(2023)Enhancing Augmented Reality Performance: An Exploration of Edge Computing and Code Offloading in Collaborative AR Systems2023 IEEE International Conference on Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE)10.1109/WETICE57085.2023.10477845(1-6)Online publication date: 14-Dec-2023
https://doi.org/10.1109/WETICE57085.2023.10477845
Khan TZhu TDownes TCheng LKass NAndrews EBiehl J(2023)Understanding Effects of Visual Feedback Delay in AR on Fine Motor Surgical TasksIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332021429:11(4697-4707)Online publication date: 3-Oct-2023
https://dl.acm.org/doi/10.1109/TVCG.2023.3320214
Sossalla PSteinke JVielhaus CHofer JRischke JFitzek F(2023)Remote Controlling Mobile Robots with Adaptive Video Streaming and Augmented Reality2023 IEEE International Black Sea Conference on Communications and Networking (BlackSeaCom)10.1109/BlackSeaCom58138.2023.10299775(45-50)Online publication date: 4-Jul-2023
https://doi.org/10.1109/BlackSeaCom58138.2023.10299775

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents