TextPolyp: Point-Supervised Polyp Segmentation with Text Cues
Authors: 
Yiming Zhao, Yi Zhou, Yizhe Zhang, Ye Wu, Tao ZhouAuthors Info & Claims
Yiming Zhao, Yi Zhou, Yizhe Zhang, Ye Wu, Tao ZhouAuthors Info & ClaimsPages 711 - 722
Abstract
Polyp segmentation in colonoscopy images is essential for preventing Colorectal cancer (CRC). Existing polyp segmentation models often struggle with costly pixel-wise annotations. Conversely, datasets can be annotated quickly and affordably using weak labels such as points. However, utilizing sparse annotations for model training remains challenging due to the limited information. In this study, we propose a TextPolyp approach to tackle this issue by leveraging only point annotations and text cues for effective weakly-supervised polyp segmentation. Specifically, we utilize the Grounding DINO algorithm and Segment Anything Model (SAM) to generate initial pseudo-labels, which are then refined with point annotations. Furthermore, we employ a SAM-based mutual learning strategy to effectively enhance segmentation results from SAM. Additionally, we propose a Discrepancy-aware Weight Scheme (DWS) to adaptively reduce the impact of unreliable predictions from SAM. Our TextPolyp model is versatile and can seamlessly integrate with various backbones and segmentation methods. Importantly, the proposed strategies are used exclusively during training, incurring no additional computational cost during inference. Extensive experiments confirm the effectiveness of our TextPolyp approach. Our code is available at https://github.com/taozh2017/TextPolyp.
References
[1]
Bernal, J., Sánchez, F.J., Fernández-Esparrach, G., Gil, D., Rodríguez, C., Vilariño, F.: WM-DOVA maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Computerized Medical Imaging and Graphics 43, 99–111 (2015)
[2]
Biswas, R.: Polyp-sam++: Can a text guided sam perform better for polyp segmentation? arXiv preprint arXiv:2308.06623 (2023)
[3]
Bui, N.T., Hoang, D.H., Nguyen, Q.T., Tran, M.T., Le, N.: Meganet: Multi-scale edge-guided attention network for weak boundary polyp segmentation. In: Proceedings of the IEEE Winter Conference on Applications of Computer Vision. pp. 7985–7994 (2024)
[4]
Chen, Z., Tian, Z., Zhu, J., Li, C., Du, S.: C-cam: Causal cam for weakly supervised semantic segmentation on medical image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 11676–11685 (2022)
[5]
Dong, B., Wang, W., Fan, D.P., Li, J., Fu, H., Shao, L.: Polyp-PVT: Polyp segmentation with pyramidvision transformers. CAAI Artificial Intelligence Research (2023)
[6]
Fan, D.P., Ji, G.P., Zhou, T., Chen, G., Fu, H., Shen, J., Shao, L.: Pranet: Parallel reverse attention network for polyp segmentation. In: International Conference on Medical Image Computing and Computer-assisted Intervention. pp. 263–273. Springer (2020)
[7]
Gao SH, Cheng MM, Zhao K, Zhang XY, Yang MH, and Torr P Res2net: A new multi-scale backbone architecture IEEE Transactions on Pattern Analysis and Machine Intelligence 2021 43 2 652-662
[8]
Gao, S., Zhang, W., Wang, Y., Guo, Q., Zhang, C., He, Y., Zhang, W.: Weakly-supervised salient object detection using point supervision. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36, pp. 670–678 (2022)
[9]
Godard, C., Mac Aodha, O., Brostow, G.J.: Unsupervised monocular depth estimation with left-right consistency. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 270–279 (2017)
[10]
He, C., Li, K., Zhang, Y., Xu, G., Tang, L., Zhang, Y., Guo, Z., Li, X.: Weakly-supervised concealed object segmentation with sam-based pseudo labeling and multi-scale feature grouping. Advances in Neural Information Processing Systems (2023)
[11]
He, R., Dong, Q., Lin, J., Lau, R.W.: Weakly-supervised camouflaged object detection with scribble annotations. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 781–789 (2023)
[12]
Huang, Z., Liu, H., Zhang, H., Xing, F., Laine, A., Angelini, E., Hendon, C., Gan, Y.: Push the boundary of sam: A pseudo-label correction framework for medical segmentation. arXiv preprint arXiv:2308.00883 (2023)
[13]
Jha, D., Smedsrud, P.H., Riegler, M.A., Halvorsen, P., de Lange, T., Johansen, D., Johansen, H.D.: Kvasir-seg: A segmented polyp dataset. In: MultiMedia Modeling. pp. 451–462. Springer (2020)
[14]
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. arXiv preprint arXiv:2304.02643 (2023)
[15]
Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Li, C., Yang, J., Su, H., Zhu, J., et al.: Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 (2023)
[16]
Liu X, Yuan Q, Gao Y, He K, Wang S, Tang X, Tang J, and Shen D Weakly supervised segmentation of COVID19 infection with scribble annotation on CT images Pattern Recognition 2022 122
[17]
Pan X, Ma C, Mu Y, and Bi M Glsnet: A global guided local feature stepwise aggregation network for polyp segmentation Biomedical Signal Processing and Control 2024 87
[18]
Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-assisted Intervention. pp. 234–241. Springer (2015)
[19]
Silva J, Histace A, Romain O, Dray X, and Granado B Toward embedded detection of polyps in WCE images for early diagnosis of colorectal cancer International Journal of Computer Assisted Radiology and Surgery 2014 9 2 283-293
[20]
Tajbakhsh N, Gurudu SR, and Liang J Automated polyp detection in colonoscopy videos using shape and context information IEEE Transactions on Medical Imaging 2015 35 2 630-644
[21]
Vázquez D, Bernal J, Sánchez FJ, Fernández-Esparrach G, López AM, Romero A, Drozdzal M, and Courville A A benchmark for endoluminal scene segmentation of colonoscopy images Journal of Healthcare Engineering 2017 2017 1 4037190
[22]
Wang, W., Xie, E., Li, X., Fan, D.P., Song, K., Liang, D., Lu, T., Luo, P., Shao, L.: Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 568–578 (2021)
[23]
Wei, J., Hu, Y., Cui, S., Zhou, S.K., Li, Z.: Weakpolyp: You only look bounding box for polyp segmentation. In: International Conference on Medical Image Computing and Computer-assisted Intervention. pp. 757–766. Springer (2023)
[24]
Yu, S., Zhang, B., Xiao, J., Lim, E.G.: Structure-consistent weakly supervised salient object detection with local saliency coherence. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 35, pp. 3234–3242 (2021)
[25]
Yue G, Zhuo G, Yan W, Zhou T, Tang C, Yang P, and Wang T Boundary uncertainty aware network for automated polyp segmentation Neural Networks 2024 170 390-404
[26]
Zhang, J., Yu, X., Li, A., Song, P., Liu, B., Dai, Y.: Weakly-supervised salient object detection via scribble annotations. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 12546–12555 (2020)
[27]
Zhao, X., Zhang, L., Lu, H.: Automatic polyp segmentation via multi-scale subtraction network. In: International Conference on Medical Image Computing and Computer-assisted Intervention. pp. 120–130. Springer (2021)
[28]
Zhou, T., Zhang, Y., Zhou, Y., Wu, Y., Gong, C.: Can sam segment polyps? arXiv preprint arXiv:2304.07583 (2023)
[29]
Zhou T, Zhou Y, He K, Gong C, Yang J, Fu H, and Shen D Cross-level feature aggregation network for polyp segmentation Pattern Recognition 2023 140
Index Terms
- TextPolyp: Point-Supervised Polyp Segmentation with Text Cues
Index terms have been assigned to the content through auto-classification.
Recommendations
Colorectal polyp segmentation based on geodesic active contours with a shape-prior model
MICCAI'10: Proceedings of the Second international conference on Virtual Colonoscopy and Abdominal Imaging: computational challenges and clinical opportunitiesAutomated polyp segmentation is important both in measuring polyp size and in improving polyp detection performance in CTC. We present a polyp segmentation method that is based on the combination of geodesic active contours and a shape-prior model of ...
M3C-Polyp: Mixed Momentum Model Committee for Improved Semi-Supervised Learning in Polyp Segmentation
WSSE '23: Proceedings of the 2023 5th World Symposium on Software EngineeringWe propose a novel approach for semi-supervised learning (SSL) in polyp segmentation using a Mixed Momentum Model Committee. Our method addresses limited labeled data and uncertainty estimation challenges. The committee M3C consists of K momentum models ...
Semi-supervised Spatial Temporal Attention Network for Video Polyp Segmentation
Medical Image Computing and Computer Assisted Intervention – MICCAI 2022AbstractDeep learning-based polyp segmentation approaches have achieved great success in image datasets. However, the frame-by-frame annotation of polyp videos requires a large amount of workload, which limits the application of polyp segmentation ...
Comments
Information & Contributors
Information
Published In
Oct 2024
847 pages
ISBN:978-3-031-72119-9
DOI:10.1007/978-3-031-72120-5
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 07 October 2024
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 26 Dec 2024