Abstract
The Colonoscopic Withdrawal Time (CWT) is the time required to withdraw the endoscope during a colonoscopy procedure. Estimating the CWT has several applications, including as a performance metric for gastroenterologists, and as an augmentation to polyp detection systems. We present a method for estimating the CWT directly from colonoscopy video based on three separate modules: egomotion computation; depth estimation; and anatomical landmark classification. Features are computed based on the modules’ outputs, which are then used to classify each frame as representing forward, stagnant, or backward motion. This allows for the optimal detection of the change points between these phases based on efficient maximization of the likelihood; from which the CWT follows directly. We collect a dataset consisting of 788 videos of colonoscopy procedures, with the CWT for each annotated by gastroenterologists. Our algorithm achieves a mean error of 1.20 min, which nearly matches the inter-rater disagreement of 1.17 min.
V. Dashinsky and I. Shimshoni—Work performed while at Alphabet.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Cheng et al. [10] studied the effect of detecting polyps during colonoscopic insertion and found it did not improve ADR.
References
Cancer Facts & Figures (2019). https://www.cancer.org/research/cancer-facts-statistics/all-cancer-facts-figures/cancer-facts-figures-2019.html. Accessed 26 Nov 2019
Colorectal Cancer Fact Sheet (2018). https://gco.iarc.fr/today/data/factsheets/cancers/10_8_9-Colorectum-fact-sheet.pdf. Accessed 08 Jan 2020
Aghanouri, M., Ghaffari, A., Serej, N.D., Rabbani, H., Adibi, P.: New image-guided method for localisation of an active capsule endoscope in the stomach. IET Image Process. 13(12), 2321–2327 (2019)
Almalioglu, Y., Saputra, M.R.U., de Gusmao, P.P., Markham, A., Trigoni, N.: GANVO: unsupervised deep monocular visual odometry and depth estimation with generative adversarial networks. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 5474–5480. IEEE (2019)
Ballesteros, C., Trujillo, M., Mazo, C., Chaves, D., Hoyos, J.: Automatic classification of non-informative frames in colonoscopy videos using texture analysis. In: Beltrán-Castañón, C., Nyström, I., Famili, F. (eds.) CIARP 2016. LNCS, vol. 10125, pp. 401–408. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-52277-7_49
Bar, O., et al.: Impact of data on generalization of AI for surgical intelligence applications. Sci. Rep. 10(1), 1–12 (2020)
Barclay, R.L., Vicari, J.J., Doughty, A.S., Johanson, J.F., Greenlaw, R.L.: Colonoscopic withdrawal times and adenoma detection during screening colonoscopy. New Engl. J. Med. 355(24), 2533–2541 (2006)
Casser, V., Pirk, S., Mahjourian, R., Angelova, A.: Depth prediction without the sensors: leveraging structure for unsupervised learning from monocular videos. In: Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-2019) (2019)
Casser, V., Pirk, S., Mahjourian, R., Angelova, A.: Unsupervised monocular depth and ego-motion learning with structure and semantics. In: CVPR Workshop on Visual Odometry and Computer Vision Applications Based on Location Cues (VOCVALC) (2019)
Cheng, C.L., et al.: Comparison of polyp detection during both insertion and withdrawal versus only withdrawal of colonoscopy: a prospective randomized trial. J. Gastroent. Hepatol. 34(8), 1377–1383 (2019)
Cho, M., Kim, J.H., Hong, K.S., Kim, J.S., Kong, H.J., Kim, S.: Identification of cecum time-location in a colonoscopy video by deep learning analysis of colonoscope movement. PeerJ 7 (2019). https://doi.org/10.7717/peerj.7256
Cho, M., Kim, J.H., Kong, H.J., Hong, K.S., Kim, S.: A novel summary report of colonoscopy: timeline visualization providing meaningful colonoscopy video information. Int. J. Colorectal Dis. 33(5), 549–559 (2018). https://doi.org/10.1007/s00384-018-2980-3
Czempiel, T., et al.: TeCNO: surgical phase recognition with multi-stage temporal convolutional networks. arXiv preprint arXiv:2003.10751 (2020)
Dietterich, T.G., Lathrop, R.H., Lozano-Pérez, T.: Solving the multiple instance problem with axis-parallel rectangles. Artif. Intell. 89(1–2), 31–71 (1997)
Farha, Y.A., Gall, J.: MS-TCN: multi-stage temporal convolutional network for action segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3575–3584 (2019)
Freedman, D., et al.: Detecting deficient coverage in colonoscopies. IEEE Trans. Med. Imaging 39(11), 3451–3462 (2020). https://doi.org/10.1109/TMI.2020.2994221
Friedman, J.H.: Greedy function approximation: a gradient boosting machine. Ann. Stat., 1189–1232 (2001)
Friedman, J.H.: Stochastic gradient boosting. Comput. Stat. Data Anal. 38(4), 367–378 (2002)
Garg, R., B.G., V.K., Carneiro, G., Reid, I.: Unsupervised CNN for single view depth estimation: geometry to the rescue. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 740–756. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46484-8_45
Gordon, A., Li, H., Jonschkowski, R., Angelova, A.: Depth from videos in the wild: unsupervised monocular depth learning from unknown cameras. In: Proceedings of the IEEE International Conference on Computer Vision (2019)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Horn, B.K., Schunck, B.G.: Determining optical flow. In: Techniques and Applications of Image Understanding, vol. 281, pp. 319–331. International Society for Optics and Photonics (1981)
Kaminski, M.F.: Increased rate of adenoma detection associates with reduced risk of colorectal cancer and death. Gastroenterology 153(1), 98–105 (2017)
Lee, S.H., et al.: An adequate level of training for technical competence in screening and diagnostic colonoscopy: a prospective multicenter evaluation of the learning curve. Gastrointest. Endosc. 67(4), 683–689 (2008)
Lee, T., et al.: Longer mean colonoscopy withdrawal time is associated with increased adenoma detection: evidence from the Bowel Cancer Screening Programme in England. Endoscopy 45(01), 20–26 (2013)
Li, R., Wang, S., Long, Z., Gu, D.: UndeepVO: monocular visual odometry through unsupervised deep learning. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 7286–7291. IEEE (2018)
Li, Y., Ding, A., Cao, Y., Liu, B., Chen, S., Liu, X.: Detection of endoscope withdrawal time in colonoscopy videos. In: IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 67–74 (2021)
Mahjourian, R., Wicke, M., Angelova, A.: Unsupervised learning of depth and ego-motion from monocular video using 3D geometric constraints. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5667–5675 (2018)
Pinheiro, G., Coelho, P., Salgado, M., Oliveira, H.P., Cunha, A.: Deep homography based localization on videos of endoscopic capsules. In: 2018 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 724–727. IEEE (2018)
Scaramuzza, D., Fraundorfer, F.: Visual odometry [tutorial]. IEEE Rob. Autom. Maga. 18(4), 80–92 (2011)
Scaramuzza, D., Fraundorfer, F.: Visual odometry part ii: matching, robustness, optimization and applications. IEEE Rob. Autom. Maga. 19(2), 78–90 (2012)
Simmons, D.T., et al.: Impact of endoscopist withdrawal speed on polyp yield: implications for optimal colonoscopy withdrawal time. Aliment. Pharmacol. Therapeut. 24(6), 965–971 (2006)
Twinanda, A.P., Shehata, S., Mutter, D., Marescaux, J., De Mathelin, M., Padoy, N.: EndoNet: a deep architecture for recognition tasks on laparoscopic videos. IEEE Trans. Med. Imaging 36(1), 86–97 (2016)
Ummenhofer, B., et al.: DeMoN: depth and motion network for learning monocular stereo. In: CVPR (2017)
Urban, G., et al.: Deep learning localizes and identifies polyps in real time with 96% accuracy in screening colonoscopy. Gastroenterology 155(4), 1069–1078 (2018)
Wang, M., Shi, Q., Song, S., Hu, C., Meng, M.Q.H.: A novel relative position estimation method for capsule robot moving in gastrointestinal tract. Sensors 19(12), 2746 (2019)
Wang, P., et al.: Development and validation of a deep-learning algorithm for the detection of polyps during colonoscopy. Nat. Biomed. Eng. 2(10), 741–748 (2018)
Wang, S., Clark, R., Wen, H., Trigoni, N.: DeepVO: towards end-to-end visual odometry with deep recurrent convolutional neural networks. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 2043–2050. IEEE (2017)
Yang, N., Wang, R., Stuckler, J., Cremers, D.: Deep virtual stereo odometry: leveraging deep depth prediction for monocular direct sparse odometry. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 817–833 (2018)
Yin, Z., Shi, J.: GeoNet: unsupervised learning of dense depth, optical flow and camera pose. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1983–1992 (2018)
Young, Y.G., et al.: Colonoscopic withdrawal time and adenoma detection in the right colon. Medicine 97 (2018)
Zhan, H., Garg, R., Saroj Weerasekera, C., Li, K., Agarwal, H., Reid, I.: Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 340–349 (2018)
Zhou, H., Ummenhofer, B., Brox, T.: DeepTAM: deep tracking and mapping. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 822–838 (2018)
Zhou, T., Brown, M., Snavely, N., Lowe, D.G.: Unsupervised learning of depth and ego-motion from video. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1851–1858 (2017)
Acknowledgment
The authors would like to thank Gaddi Menahem and Yaron Frid of Orpheus Medical Ltd. for helping in the provision of data.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Katzir, L. et al. (2023). Estimating Withdrawal Time in Colonoscopies. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds) Computer Vision – ECCV 2022 Workshops. ECCV 2022. Lecture Notes in Computer Science, vol 13803. Springer, Cham. https://doi.org/10.1007/978-3-031-25066-8_28
Download citation
DOI: https://doi.org/10.1007/978-3-031-25066-8_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-25065-1
Online ISBN: 978-3-031-25066-8
eBook Packages: Computer ScienceComputer Science (R0)