article

Virtual annotations of the surgical field through an augmented reality transparent display

Authors:

Daniel Andersen,

Maria Eugenia Cabrera,

Aditya Shanghavi,

Juan WachsAuthors Info & Claims

The Visual Computer: International Journal of Computer Graphics, Volume 32, Issue 11

Pages 1481 - 1498

https://doi.org/10.1007/s00371-015-1135-6

Published: 01 November 2016 Publication History

Abstract

Existing telestrator-based surgical telementoring systems require a trainee surgeon to shift focus frequently between the operating field and a nearby monitor to acquire and apply instructions from a remote mentor. We present a novel approach to surgical telementoring where annotations are superimposed directly onto the surgical field using an augmented reality (AR) simulated transparent display. We present our first steps towards realizing this vision, using two networked conventional tablets to allow a mentor to remotely annotate the operating field as seen by a trainee. Annotations are anchored to the surgical field as the trainee tablet moves and as the surgical field deforms or becomes occluded. The system is built exclusively from compact commodity-level components--all imaging and processing are performed on the two tablets.

References

[1]

Agarwal, R., Levinson, A.W., Allaf, M., Makarov, D.V., Nason, A., Su, L.M.: The roboconsultant: telementoring and remote presence in the operating room during minimally invasive urologic surgeries using a novel mobile robotic interface. Urology 70(5), 970---974 (2007)

[2]

Amazon.com, I.: Amazon Fire Phone (2014). http://www.amazon.com/firephone

[3]

Ballantyne, G.H.: Robotic surgery, telerobotic surgery, telepresence, and telementoring. Surg. Endosc. Interv. Tech. 16(10), 1389---1402 (2002)

[4]

Bariă¿ević, D., Höllerer, T., Sen, P., Turk, M.: User-perspective augmented reality magic lens from gradients. In: Proceedings of the 20th ACM Symposium on Virtual Reality Software and Technology, pp. 87---96. ACM (2014)

[5]

Bashshur, R.L.: On the definition and evaluation of telemedicine. Telemed. J. 1(1), 19---30 (1995).

[6]

Bogen, E.M., Augestad, K.M., Patel, H.R., Lindsetmo, R.O.: Telementoring in education of laparoscopic surgeons: An emerging technology. World J. Gastrointest. Endosc. 6(5), 148---155 (2014). http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4024487/

[7]

Chou, W., Wang, T., Zhang, Y.: Augmented reality based preoperative planning for robot assisted tele-neurosurgery. In: Systems, Man and Cybernetics, 2004 IEEE International Conference on, vol. 3, pp. 2901---2906 vol. 3 (2004)

[8]

Ereso, A.Q., Garcia, P., Tseng, E., Gauger, G., Kim, H., Dua, M.M., Victorino, G.P., Guy, T.S.: Live transference of surgical subspecialty skills using telerobotic proctoring to remote general surgeons. J. Am. College Surg. 211(3), 400---411 (2010)

[9]

Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381---395 (1981)

Digital Library

[10]

Google: ATAP Project Tango (2014). https://www.google.com/atap/projecttango/

[11]

Guo, Y., Henao, O., Jackson, T., Quereshy, F., Okrainec, A.: Commercial videoconferencing for use in telementoring laparoscopic surgery. Med. Meets Virtual Real. 21: NextMed/MMVR21 196, 147 (2014)

[12]

Itseez: OpenCV (2014). http://opencv.org/

[13]

Khronos: OpenGL ES--the standard for embedded accelerated 3D graphics (2014). https://www.khronos.org/opengles/

[14]

Loescher, T., Lee, S.Y., Wachs, J.P.: An augmented reality approach to surgical telementoring. In: Systems, Man and Cybernetics (SMC), 2014 IEEE International Conference on, pp. 2341---2346. IEEE (2014)

[15]

Marescaux, J., Diana, M.: Robotics and remote surgery: Next step. In: K.C. Kim (ed.) Robotics in General Surgery, pp. 479---484-. Springer New York (2014)

[16]

Marescaux, J., Rubino, F.: Telesurgery, telementoring, virtual surgery, and telerobotics. Curr. Urol. Rep. 4(2), 109---113 (2003)

[17]

Occipital, I.: The Structure Sensor is the first 3D sensor for mobile devices (2014). http://structure.io/

[18]

Ponce, B.A., Jennings, J.K., Clay, T.B., May, M.B., Huisingh, C., Sheppard, E.D.: Telementoring: Use of augmented reality in orthopaedic education. J. Bone Jt. Surg. 96(10), e84 (2014). http://jbjs.org/content/96/10/e84.abstract

[19]

Rosten, E., Drummond, T.: Machine learning for high-speed corner detection. In: Computer VisionECCV 2006, pp. 430---443. Springer (2006)

Digital Library

[20]

Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: ORB: an efficient alternative to SIFT or SURF. In: Computer Vision (ICCV), 2011 IEEE International Conference on, pp. 2564---2571. IEEE (2011)

Digital Library

[21]

Satava, R.: Virtual endoscopy. Surg. Endosc. 10(2), 173---174 (1996)

[22]

Schulam, P., Docimo, S., Saleh, W., Breitenbach, C., Moore, R., Kavoussi, L.: Telesurgical mentoring. Surg. Endosc. 11(10), 1001---1005 (1997)

[23]

Shenai, M.B., Dillavou, M., Shum, C., Ross, D., Tubbs, R.S., Shih, A., Guthrie, B.L.: Virtual interactive presence and augmented reality (VIPAR) for remote surgical assistance. Neurosurgery 68, ons200-ons207 (2011)

[24]

Smurro, J.P., Reina, G.A., L'esperance, J.O.: System and method for surgical telementoring and training with virtualized telestration and haptic holograms, including metadata tagging, encapsulation and saving multi-modal streaming medical imagery together with multi-dimensional {4-d} virtual mesh and multi-sensory annotation in standard file formats used for digital imaging and communications in medicine (dicom). US Patent App. 14/138,045, Google Patents (2014). https://www.google.com/patents/US20140176661

[25]

Tomioka, M., Ikeda, S., Sato, K.: Approximated user-perspective rendering in tablet-based augmented reality. In: Mixed and Augmented Reality (ISMAR), 2013 IEEE International Symposium on, pp. 21---28. IEEE (2013)

[26]

Treter, S., Perrier, N., Sosa, J.A., Roman, S.: Telementoring: a multi-institutional experience with the introduction of a novel surgical approach for adrenalectomy. Ann. Surg. Oncol. 20(8), 2754---2758 (2013)

[27]

Unuma, Y., Niikura, T., Komuro, T.: See-through mobile ar system for natural 3d interaction. In: Proceedings of the companion publication of the 19th international conference on Intelligent User Interfaces, pp. 17---20. ACM (2014)

Digital Library

[28]

Vera, A.M., Russo, M., Mohsin, A., Tsuda, S.: Augmented reality telementoring (ART) platform: a randomized controlled trial to assess the efficacy of a new surgical education technology. Surg. Endosc. 28(12), 3467---3472 (2014)

Cited By

Ye BFujimoto YSawabe TKanbara MKato H(2024)Artifact reduction in lenslet array near-eye displaysThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-023-02804-240:2(601-618)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1007/s00371-023-02804-2
Wecker DYigitbas E(2023)Minimizing Eye Movements and Distractions in Head-Mounted Augmented Reality through Eye-Gaze AdaptivenessProceedings of the 2023 ACM Symposium on Spatial User Interaction10.1145/3607822.3614524(1-12)Online publication date: 13-Oct-2023
https://dl.acm.org/doi/10.1145/3607822.3614524
Kong LYang TXie LXu DHe K(2022)Cascade connection-based channel attention network for bidirectional medical image registrationThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-022-02678-w39:11(5527-5545)Online publication date: 2-Oct-2022
https://dl.acm.org/doi/10.1007/s00371-022-02678-w
Show More Cited By

Index Terms

Virtual annotations of the surgical field through an augmented reality transparent display
1. Human-centered computing
  1. Human computer interaction (HCI)

Index terms have been assigned to the content through auto-classification.

Recommendations

Depth cues in augmented reality for training of robot-assisted minimally invasive surgery
AcademicMindtrek '17: Proceedings of the 21st International Academic Mindtrek Conference

Training of robot-assisted minimally invasive surgery often includes supervised practice with a robotic surgical system. In this case, augmented reality can improve the communication between instructor and trainee, for example, by allowing the ...
An augmented reality system for surgical pre-operative planning
BEBI'08: Proceedings of the 1st WSEAS international conference on Biomedical electronics and biomedical informatics

Augmented Reality technology can be used in minimally invasive surgery to increase the physician's view by showing information gathered from patient medical images. The developed AR system can improve the performances of a surgeon involved in a ...
Augmented Reality for Supporting Real Time Telementoring: an Exploratory Study Applied to Ultrasonography
ICMHI '18: Proceedings of the 2nd International Conference on Medical and Health Informatics

Background: Telementoring is a real time telemedicine modality that allows a specialist to remotely assist a less expert operator. Videoconferencing is the main means for implementing telementoring applications and has been used in a wide range of ...

Comments

Information & Contributors

Information

Published In

cover image The Visual Computer: International Journal of Computer Graphics

The Visual Computer: International Journal of Computer Graphics Volume 32, Issue 11

November 2016

138 pages

ISSN:0178-2789

Issue’s Table of Contents

Copyright © Copyright © 2016 Springer-Verlag Berlin Heidelberg.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 01 November 2016

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 16 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ye BFujimoto YSawabe TKanbara MKato H(2024)Artifact reduction in lenslet array near-eye displaysThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-023-02804-240:2(601-618)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1007/s00371-023-02804-2
Wecker DYigitbas E(2023)Minimizing Eye Movements and Distractions in Head-Mounted Augmented Reality through Eye-Gaze AdaptivenessProceedings of the 2023 ACM Symposium on Spatial User Interaction10.1145/3607822.3614524(1-12)Online publication date: 13-Oct-2023
https://dl.acm.org/doi/10.1145/3607822.3614524
Kong LYang TXie LXu DHe K(2022)Cascade connection-based channel attention network for bidirectional medical image registrationThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-022-02678-w39:11(5527-5545)Online publication date: 2-Oct-2022
https://dl.acm.org/doi/10.1007/s00371-022-02678-w
Gasques DJohnson JSharkey TFeng YWang RXu ZZavala EZhang YXie WZhang XDavis KYip MWeibel NKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)ARTEMIS: A Collaborative Mixed-Reality System for Immersive Surgical TelementoringProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445576(1-14)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445576
Ferdous HHoang TJoukhadar ZReinoso MVetere FKelly DRemedios LBrewster SFitzpatrick GCox AKostakos V(2019)"What's Happening at that Hip?"Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300464(1-12)Online publication date: 2-May-2019
https://dl.acm.org/doi/10.1145/3290605.3300464
Krishnakumar KIndira Gandhi S(2019)Video stitching based on multi-view spatiotemporal feature points and grid-based matchingThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-019-01780-w36:9(1837-1846)Online publication date: 7-Dec-2019
https://dl.acm.org/doi/10.1007/s00371-019-01780-w
Bailly CLeitner FNigay L(2019)Head-Controlled Menu in Mixed Reality with a HMDHuman-Computer Interaction – INTERACT 201910.1007/978-3-030-29390-1_22(395-415)Online publication date: 2-Sep-2019
https://dl.acm.org/doi/10.1007/978-3-030-29390-1_22
Zhou TWachs J(2018)Early prediction for physical human robot collaboration in the operating roomAutonomous Robots10.1007/s10514-017-9670-942:5(977-995)Online publication date: 1-Jun-2018
https://dl.acm.org/doi/10.1007/s10514-017-9670-9

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents