Abstract
Purpose:
In medical research, deep learning models rely on high-quality annotated data, a process often laborious and time-consuming. This is particularly true for detection tasks where bounding box annotations are required. The need to adjust two corners makes the process inherently frame-by-frame. Given the scarcity of experts’ time, efficient annotation methods suitable for clinicians are needed.
Methods:
We propose an on-the-fly method for live video annotation to enhance the annotation efficiency. In this approach, a continuous single-point annotation is maintained by keeping the cursor on the object in a live video, mitigating the need for tedious pausing and repetitive navigation inherent in traditional annotation methods. This novel annotation paradigm inherits the point annotation’s ability to generate pseudo-labels using a point-to-box teacher model. We empirically evaluate this approach by developing a dataset and comparing on-the-fly annotation time against traditional annotation method.
Results:
Using our method, annotation speed was \(3.2\times \) faster than the traditional annotation technique. We achieved a mean improvement of \(6.51 \pm 0.98\) AP@50 over conventional method at equivalent annotation budgets on the developed dataset.
Conclusion:
Without bells and whistles, our approach offers a significant speed-up in annotation tasks. It can be easily implemented on any annotation platform to accelerate the integration of deep learning in video-based medical research.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Buch VH, Ahmed I, Maruthappu M (2018) Artificial intelligence in medicine: current trends and future possibilities. Br J Gen Pract 68(668):143–144
Mascagni P, Alapatt D, Sestini L, Altieri MS, Madani A, Watanabe Y, Alseidi A, Redan JA, Alfieri S, Costamagna G et al (2022) Computer vision in surgery: from potential to clinical value. npj Digital Med 5(1):163
Zhang H, Li F, Liu S, Zhang L, Su H, Zhu J, Ni L, Shum H (2022) Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arxiv 2022. arXiv preprint arXiv:2203.03605
Lyu C, Zhang W, Huang H, Zhou Y, Wang Y, Liu Y, Zhang S, Chen K (2022) RTMDet: an empirical study of designing real-time object detectors
Barua I, Vinsard DG, Jodal HC, Løberg M, Kalager M, Holme Ø, Misawa M, Bretthauer M, Mori Y (2020) Artificial intelligence for polyp detection during colonoscopy: a systematic review and meta-analysis. Endoscopy 53(03):277–284
Twinanda AP, Shehata S, Mutter D, Marescaux J, De Mathelin M, Padoy N (2016) Endonet: a deep architecture for recognition tasks on laparoscopic videos. IEEE Trans Med Imaging 36(1):86–97
Srivastav V, Issenhuth T, Kadkhodamohammadi A, Mathelin M, Gangi A, Padoy N (2018) Mvor: a multi-view rgb-d operating room dataset for 2d and 3d human pose estimation. arXiv preprint arXiv:1808.08180
Krenzer A, Makowski K, Hekalo A, Fitting D, Troya J, Zoller WG, Hann A, Puppe F (2022) Fast machine learning annotation in the medical domain: a semi-automated video annotation tool for gastroenterologists. Biomed Eng Online 21(1):1–23
Kirillov A, Mintun E, Ravi N, Mao H, Rolland C, Gustafson L, Xiao T, Whitehead S, Berg AC, Lo W-Y, et al (2023) Segment anything. arXiv preprint arXiv:2304.02643
Chen L, Yang T, Zhang X, Zhang W, Sun J (2021) Points as queries: weakly semi-supervised object detection by points. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 8823–8832
Zhang S, Yu Z, Liu L, Wang X, Zhou A, Chen K (2022) Group r-cnn for weakly semi-supervised object detection with points. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 9417–9426
Rother A, Niemann U, Hielscher T, Völzke H, Ittermann T, Spiliopoulou M (2021) Assessing the difficulty of annotating medical data in crowd working with help of experiments. PLoS ONE 16(7):0254764
Heim E, Roß T, Seitel A, März K, Stieltjes B, Eisenmann M, Lebert J, Metzger J, Sommer G, Sauter AW et al (2018) Large-scale medical image annotation with crowd-powered algorithms. J Med Imaging 5(3):034002–034002
Jeong J, Lee S, Kim J, Kwak N (2019) Consistency-based semi-supervised learning for object detection. Adv Neural Inf Process Syst, 32
Liu Y-C, Ma C-Y, He Z, Kuo C-W, Chen K, Zhang P, Wu B, Kira Z, Vajda P (2021) Unbiased teacher for semi-supervised object detection. arXiv preprint arXiv:2102.09480
Wang Z, Li Y, Guo Y, Fang L, Wang S (2021) Data-uncertainty guided multi-phase learning for semi-supervised object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 4568–4577
LaBonte T, Song Y, Wang X, Vineet V, Joshi N (2023) Scaling novel object detection with weakly supervised detection transformers. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp 85–96
Tang P, Wang X, Bai S, Shen W, Bai X, Liu W, Yuille A (2018) PCL: proposal cluster learning for weakly supervised object detection. IEEE Trans Pattern Anal Mach Intell 42(1):176–191
Vardazaryan A, Mutter D, Marescaux J, Padoy N (2018) Weakly-supervised learning for tool localization in laparoscopic videos. In: Intravascular imaging and computer assisted stenting and large-scale annotation of biomedical data and expert label synthesis: 7th joint international workshop, CVII-STENT 2018 and third international workshop, LABELS 2018, held in conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Proceedings 3, pp 169–179. Springer
Kim J, Kim HJ, Kim C, Lee JH, Kim KW, Park YM, Kim HW, Ki SY, Kim YM, Kim WH (2021) Weakly-supervised deep learning for ultrasound diagnosis of breast cancer. Sci Rep 11(1):24382
Ouyang J, Chen L, Li GY, Balaraju N, Patil S, Mehanian C, Kulhare S, Millin R, Gregory KW, Gregory CR et al (2023) Weakly semi-supervised detection in lung ultrasound videos. In: International conference on information processing in medical imaging, pp 195–207. Springer
Carion N, Massa F, Synnaeve G, Usunier N, Kirillov A, Zagoruyko S (2020) End-to-end object detection with transformers. In: European conference on computer vision, pp 213–229. Springer
Ren S, He K, Girshick R, Sun J (2015) Faster r-cnn: towards real-time object detection with region proposal networks. Adv Neural Inf Process Syst 28
Lin T-Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft coco: common objects in context. In: Computer vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6–12, 2014, Proceedings, Part V 13, pp 740–755. Springer
Chen K, Wang J, Pang J, Cao Y, Xiong Y, Li X, Sun S, Feng W, Liu Z, Xu J et al (2019) Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155
Kim DD, Chandra RS, Peng J, Wu J, Feng X, Atalay M, Bettegowda C, Jones C, Sair H, Liao W-h et al (2023) Active learning in brain tumor segmentation with uncertainty sampling, annotation redundancy restriction, and data initialization. arXiv preprint arXiv:2302.10185
Wang K, Zhang D, Li Y, Zhang R, Lin L (2016) Cost-effective active learning for deep image classification. IEEE Trans Circuits Syst Video Technol 27(12):2591–2600
Acknowledgements
This research was conducted within the framework of the APEUS and TheraHCC 2.0 projects, which are supported by the ARC Foundation (www.fondation-arc.org). This work was also partially supported by French state funds managed within the ’Plan Investissements d’Avenir’, funded by the ANR (reference ANR-10-IAHU-02 and ANR-21-RHUS-0001 DELIVER). This work was performed using HPC resources from GENCI-IDRIS (Grant 2023-AD011013698R1).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Ethical approval
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards (ID-RCB 2020-A02949-30; NCT04802954).
Informed consent
Informed consent was obtained from all individual participants included in the study.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Meyer, A., Mazellier, JP., Dana, J. et al. On-the-fly point annotation for fast medical video labeling. Int J CARS 19, 1093–1101 (2024). https://doi.org/10.1007/s11548-024-03098-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11548-024-03098-y