Abstract
We present a system for marking, tracking and visually augmenting a deformable surgical site by the robust automatic detection of natural landmarks (image features) in laparoscopic surgery. In our system, the surgeon first selects a frame containing an organ of interest, and this is used by our system both to detect every instance of the organ in a laparoscopic video feed, and to recover the nonrigid deformations. The system then augments the video with customizable visual information such as the location of hidden or weakly visible structures (cysts, vessels, etc), or planned incision points, acquired from pre-operative or intra-operative data. Frame-rate organ detection is performed via a novel procedure that matches the current frame to the reference frame. Because laparoscopic images are known to be extremely difficult to match, we propose to use Shape-from-Shading and conformal flattening to cancel out much of the variation in appearance due to perspective foreshortening, and we then apply robust matching to the flattened surfaces. Experiments show robust tracking and detection results on a laparoscopic procedure with the uterus as target organ. As our system detects the organ in every frame, it is not impaired by target loss, contrary to most previous methods.
This research has been supported by Prof. Richard Hartley’s Chair of Excellence grant from Région Auvergne, France.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Collins, T., Bartoli, A.: Towards live monocular 3d laparoscopy using shading and specularity information. In: IPCAI (2012)
Fischler, M., Bolles, R.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM 24(6) (June 1981)
Gu, X., Yau, S.: Surface classification using conformal structures. In: Proceedings. Ninth IEEE International Conference on Computer Vision, vol. 1, pp. 701–708 (2003)
Haker, S., Angenent, S., Tannenbaum, A., Kikinis, R., Sapiro, G., Halle, M.: Conformal surface parameterization for texture mapping. IEEE Transactions on Visualization and Computer Graphics 6(2), 181–189 (2000)
Horn, B.K.P.: Closed-form solution of absolute orientation using unit quaternions. JOSA A 4(4), 629–642 (1987)
Lévy, B., Petitjean, S., Ray, N., Maillot, J.: Least squares conformal maps for automatic texture atlas generation. In: ACM SIGGRAPH Conference Proceedings (April 2002)
Lowe, D.: Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60(2), 91–110 (2004)
Mountney, P., Lo, B., Thiemjarus, S., Stoyanov, D., Zhong-Yang, G.: A Probabilistic Framework for Tracking Deformable Soft Tissue in Minimally Invasive Surgery. In: Ayache, N., Ourselin, S., Maeder, A. (eds.) MICCAI 2007, Part II. LNCS, vol. 4792, pp. 34–41. Springer, Heidelberg (2007)
Nicolau, S., Solar, L., Mutter, D., Marescaux, J.: Augmented reality in laparoscopy surgical oncology. Surgical Oncology 20, 189–201 (2011)
Pizarro, D., Bartoli, A.: Feature-based deformable surface detection with self-occlusion reasoning. International Journal of Computer Vision (2011) (published online April 27, 2011)
Schaefer, S., McPhail, T., Warren, J.: Image deformation using moving least squares. ACM Transactions on Graphics (TOG) 25(3), 533–540 (2006)
Seiden, S.C., Barach, P.: Wrong-side/wrong-site, wrong-procedure, and wrong-patient adverse events: Are they preventable? Archives of Surgery 141(9), 931–939 (2006)
Shuhaiber, J.H.: Augmented reality in surgery. ARCH SURG 139, 170–174 (2004)
Su, L.M., Vagvolgyi, B.P., Agarwal, R., Reiley, C.E., Taylor, R.H., Hager, G.D.: Augmented reality during robot-assisted laparoscopic partial nephrectomy: Toward real-time 3d-ct to stereoscopic video registration. Journal of Urology 73(4), 896–900 (2009)
Zhang, R., Tsai, P., Cryer, J., Shah, M.: Shape-from-shading: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 21(8), 690–706 (1999)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kim, JH., Bartoli, A., Collins, T., Hartley, R. (2012). Tracking by Detection for Interactive Image Augmentation in Laparoscopy. In: Dawant, B.M., Christensen, G.E., Fitzpatrick, J.M., Rueckert, D. (eds) Biomedical Image Registration. WBIR 2012. Lecture Notes in Computer Science, vol 7359. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-31340-0_26
Download citation
DOI: https://doi.org/10.1007/978-3-642-31340-0_26
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-31339-4
Online ISBN: 978-3-642-31340-0
eBook Packages: Computer ScienceComputer Science (R0)