research-article

Segmenting medical images with limited data

Authors:

Lei ShenAuthors Info & Claims

Volume 177, Issue C

https://doi.org/10.1016/j.neunet.2024.106367

Published: 24 July 2024 Publication History

Abstract

While computer vision has proven valuable for medical image segmentation, its application faces challenges such as limited dataset sizes and the complexity of effectively leveraging unlabeled images. To address these challenges, we present a novel semi-supervised, consistency-based approach termed the data-efficient medical segmenter (DEMS). The DEMS features an encoder–decoder architecture and incorporates the developed online automatic augmenter (OAA) and residual robustness enhancement (RRE) blocks. The OAA augments input data with various image transformations, thereby diversifying the dataset to improve the generalization ability. The RRE enriches feature diversity and introduces perturbations to create varied inputs for different decoders, thereby providing enhanced variability. Moreover, we introduce a sensitive loss to further enhance consistency across different decoders and stabilize the training process. Extensive experimental results on both our own and three public datasets affirm the effectiveness of DEMS. Under extreme data shortage scenarios, our DEMS achieves 16.85% and 10.37% improvement in dice score compared with the U-Net and top-performed state-of-the-art method, respectively. Given its superior data efficiency, DEMS could present significant advancements in medical segmentation under small data regimes. The project homepage can be accessed at https://github.com/NUS-Tim/DEMS.

References

[1]

1st place solution in MICCAI 2020 TN-SCUI challenge, 2023, https://github.com/WAMAWAMA/TNSCUI2020-Seg-Rank1st. (Accessed 5 September 2023).

[2]

Abualigah Laith, Ekinci Serdar, Izci Davut, Zitar R. Abu, Modified elite opposition-based artificial hummingbird algorithm for designing FOPID controlled cruise control system, Intelligent Automation & Soft Computing (2023).

[3]

Al-Dhabyani Walid, Gomaa Mohammed, Khaled Hussien, Fahmy Aly, Dataset of breast ultrasound images, Data in Brief 28 (2020),.

[4]

Buslaev Alexander, Iglovikov Vladimir I., Khvedchenya Eugene, Parinov Alex, Druzhinin Mikhail, Kalinin Alexandr A., Albumentations: fast and flexible image augmentations, Information 11 (2) (2020) 125,.

[5]

Cao Hu, Wang Yueyue, Chen Joy, Jiang Dongsheng, Zhang Xiaopeng, Tian Qi, et al., Swin-unet: Unet-like pure transformer for medical image segmentation, in: European conference on computer vision, Springer, 2022, pp. 205–218.

[6]

Chai Lu, Wang Zidong, Chen Jianqing, Zhang Guokai, Alsaadi Fawaz E., Alsaadi Fuad E., et al., Synthetic augmentation for semantic segmentation of class imbalanced biomedical images: A data pair generative adversarial network approach, Computers in Biology and Medicine 150 (2022).

[7]

Chen Jingying, Shi Jinxin, Xu Ruyi, Dual subspace manifold learning based on GCN for intensity-invariant facial expression recognition, Pattern Recognition 148 (2024).

[8]

Chen, Xiaokang, Yuan, Yuhui, Zeng, Gang, & Wang, Jingdong (2021). Semi-supervised semantic segmentation with cross pseudo supervision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 2613–2622).

[9]

Cubuk, Ekin D., Zoph, Barret, Mane, Dandelion, Vasudevan, Vijay, & Le, Quoc V. (2019). Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 113–123).

[10]

Eaton-Rosen, Zach, Bragman, Felix, Ourselin, Sebastien, & Cardoso, M. Jorge (2018). Improving data augmentation for medical image segmentation. In Proceedings of the conference on medical imaging with deep learning.

[11]

Eisenmann Matthias, Reinke Annika, Weru Vivienn, Tizabi Minu Dietlinde, Isensee Fabian, Adler Tim J., et al., Why is the winner the best?, arXiv preprint (2023),.

[12]

Feng, Qianli, Guo, Chenqi, Benitez-Quiroz, Fabian, & Martinez, Aleix M. (2021). When do gans replicate? on the choice of dataset size. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 6701–6710).

[13]

Goodfellow Ian, Pouget-Abadie Jean, Mirza Mehdi, Xu Bing, Warde-Farley David, Ozair Sherjil, et al., Generative adversarial nets, Advances in Neural Information Processing Systems 27 (2014).

Digital Library

[14]

Guo Saidi, Liu Xiujian, Zhang Heye, Lin Qixin, Xu Lei, Shi Changzheng, et al., Causal knowledge fusion for 3D cross-modality cardiac image segmentation, Information Fusion (2023).

[15]

Guo Saidi, Zhang Heye, Gao Yifeng, Wang Hui, Xu Lei, Gao Zhifan, et al., Survival prediction of heart failure patients using motion-based analysis method, Computer Methods and Programs in Biomedicine 236 (2023).

[16]

He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, & Sun, Jian (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770–778).

[17]

Hu Gang, Zheng Yixuan, Abualigah Laith, Hussien Abdelazim G., DETDO: An adaptive hybrid dandelion optimizer for engineering optimization, Advanced Engineering Informatics 57 (2023).

[18]

Iqbal Ahmed, Sharif Muhammad, UNet: A semi-supervised method for segmentation of breast tumor images using a U-shaped pyramid-dilated network, Expert Systems with Applications 221 (2023).

[19]

Isensee Fabian, Jaeger Paul F., Kohl Simon A.A., Petersen Jens, Maier-Hein Klaus H., nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation, Nature Methods 18 (2) (2021) 203–211.

[20]

Jiao Rushi, Zhang Yichi, Ding Le, Cai Rong, Zhang Jicong, Learning with limited annotations: a survey on deep semi-supervised learning for medical image segmentation, 2022, arXiv preprint arXiv:2207.14191.

[21]

Karras, Tero, Laine, Samuli, & Aila, Timo (2019). A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 4401–4410).

[22]

Krizhevsky Alex, Sutskever Ilya, Hinton Geoffrey E., Imagenet classification with deep convolutional neural networks, Communications of the ACM 60 (6) (2017) 84–90,.

[23]

Kugelman Jason, Alonso-Caneiro David, Read Scott A., Vincent Stephen J., Chen Fred K., Collins Michael J., Dual image and mask synthesis with GANs for semantic segmentation in optical coherence tomography, in: 2020 digital image computing: techniques and applications, IEEE, 2020, pp. 1–8.

[24]

Lei Tao, Zhang Dong, Du Xiaogang, Wang Xuan, Wan Yong, Nandi Asoke K., Semi-supervised medical image segmentation using adversarial consistency learning and dynamic convolution network, IEEE Transactions on Medical Imaging (2022).

[25]

Li Xiaoqiang, Wu Yuanchen, Dai Songmin, Semi-supervised medical imaging segmentation with soft pseudo-label fusion, Applied Intelligence: The International Journal of Artificial Intelligence, Neural Networks, and Complex Problem-Solving Technologies (2023) 1–13.

[26]

Li Xiaokang, Xia Menghua, Jiao Jing, Zhou Shichong, Chang Cai, Wang Yuanyuan, et al., HAL-IA: A hybrid active learning framework using interactive annotation for medical image segmentation, Medical Image Analysis (2023).

[27]

Li, Daiqing, Yang, Junlin, Kreis, Karsten, Torralba, Antonio, & Fidler, Sanja (2021). Semantic segmentation with generative models: Semi-supervised learning and strong out-of-domain generalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 8300–8311).

[28]

Li Penghui, Zhou Rui, He Jin, Zhao Shifeng, Tian Yun, A global-frequency-domain network for medical image segmentation, Computers in Biology and Medicine (2023).

[29]

Liu Zhaoshan, Lv Qiujie, Lee Chau Hung, Shen Lei, GSDA: Generative adversarial network-based semi-supervised data augmentation for ultrasound image classification, Heliyon 9 (9) (2023).

[30]

Liu Zhaoshan, Lv Qiujie, Li Yifan, Yang Ziduo, Shen Lei, MedAugment: Universal automatic data augmentation plug-in for medical image analysis, 2023, arXiv preprint arXiv:2306.17466.

[31]

Liu Zhaoshan, Lv Qiujie, Yang Ziduo, Li Yifan, Lee Chau Hung, Shen Lei, Recent progress in transformer-based medical image analysis, Computers in Biology and Medicine (2023).

[32]

Loshchilov Ilya, Hutter Frank, Sgdr: Stochastic gradient descent with warm restarts, 2016, arXiv preprint arXiv:1608.03983.

[33]

Luo Xiangde, Hu Minhao, Song Tao, Wang Guotai, Zhang Shaoting, Semi-supervised medical image segmentation via cross teaching between cnn and transformer, in: International conference on medical imaging with deep learning, PMLR, 2022, pp. 820–833.

[34]

Luo Xiangde, Liao Wenjun, Chen Jieneng, Song Tao, Chen Yinan, Zhang Shichuan, et al., Efficient semi-supervised gross target volume of nasopharyngeal carcinoma segmentation via uncertainty rectified pyramid consistency, in: Medical image computing and computer assisted intervention–MICCAI 2021: 24th international conference, Strasbourg, France, September 27–October 1, 2021, proceedings, part II 24, Springer, 2021, pp. 318–329.

[35]

Lyu Jun, Sui Bin, Wang Chengyan, Dou Qi, Qin Jing, Adaptive feature aggregation based multi-task learning for uncertainty-guided semi-supervised medical image segmentation, Expert Systems with Applications (2023).

[36]

Lyu Junyan, Zhang Yiqi, Huang Yijin, Lin Li, Cheng Pujin, Tang Xiaoying, Aadg: automatic augmentation for domain generalization on retinal image segmentation, IEEE Transactions on Medical Imaging 41 (12) (2022) 3699–3711.

[37]

Murdoch Blake, Privacy and artificial intelligence: challenges for protecting health information in a new era, BMC Medical Ethics 22 (1) (2021) 1–5.

[38]

Ouali, Yassine, Hudelot, Céline, & Tami, Myriam (2020). Semi-supervised semantic segmentation with cross-consistency training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 12674–12684).

[39]

Pandey Siddharth, Singh Pranshu Ranjan, Tian Jing, An image augmentation approach using two-stage generative adversarial network for nuclei image segmentation, Biomedical Signal Processing and Control 57 (2020).

[40]

Pedraza Lina, Vargas Carlos, Narváez Fabián, Durán Oscar, Muñoz Emma, Romero Eduardo, An open access thyroid ultrasound image database, in: 10th international symposium on medical information processing and analysis, vol. 9287, SPIE, 2015, pp. 188–193.

[41]

Qin Tiexin, Wang Ziyuan, He Kelei, Shi Yinghuan, Gao Yang, Shen Dinggang, Automatic data augmentation via deep reinforcement learning for effective kidney tumor segmentation, in: ICASSP 2020-2020 IEEE international conference on acoustics, speech and signal processing, IEEE, 2020, pp. 1419–1423.

[42]

Ronneberger Olaf, Fischer Philipp, Brox Thomas, U-net: Convolutional networks for biomedical image segmentation, in: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5–9, 2015, proceedings, part III 18, Springer, 2015, pp. 234–241.

[43]

Russell Bryan C., Torralba Antonio, Murphy Kevin P., Freeman William T., LabelMe: a database and web-based tool for image annotation, International Journal of Computer Vision 77 (2008) 157–173.

Digital Library

[44]

Tang Fenghe, Ding Jianrui, Wang Lingtao, Xian Min, Ning Chunping, Multi-level global context cross consistency model for semi-supervised ultrasound image segmentation with diffusion model, 2023, arXiv preprint arXiv:2305.09447.

[45]

Tarvainen Antti, Valpola Harri, Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results, Advances in Neural Information Processing Systems 30 (2017).

[46]

Tompson, Jonathan, Goroshin, Ross, Jain, Arjun, LeCun, Yann, & Bregler, Christoph (2015). Efficient object localization using convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 648–656).

[47]

Verma Vikas, Kawaguchi Kenji, Lamb Alex, Kannala Juho, Solin Arno, Bengio Yoshua, et al., Interpolation consistency training for semi-supervised learning, Neural Networks 145 (2022) 90–106.

[48]

Wang Ziyu, Schaul Tom, Hessel Matteo, Hasselt Hado, Lanctot Marc, Freitas Nando, Dueling network architectures for deep reinforcement learning, in: International conference on machine learning, PMLR, 2016, pp. 1995–2003.

[49]

Wang, Yongchao, Xiao, Bin, Bi, Xiuli, Li, Weisheng, & Gao, Xinbo (2023). MCF: Mutual Correction Framework for Semi-Supervised Medical Image Segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 15651–15660).

[50]

Wu Yicheng, Ge Zongyuan, Zhang Donghao, Xu Minfeng, Zhang Lei, Xia Yong, et al., Mutual consistency learning for semi-supervised medical image segmentation, Medical Image Analysis 81 (2022).

[51]

Wu Lijun, Li Juntao, Wang Yue, Meng Qi, Qin Tao, Chen Wei, et al., R-drop: Regularized dropout for neural networks, Advances in Neural Information Processing Systems 34 (2021) 10890–10905.

[52]

Wu Huisi, Liu Jiasheng, Xiao Fangyan, Wen Zhenkun, Cheng Lan, Qin Jing, Semi-supervised segmentation of echocardiography videos via noise-resilient spatiotemporal semantic calibration and fusion, Medical Image Analysis 78 (2022).

[53]

Wu Yicheng, Xu Minfeng, Ge Zongyuan, Cai Jianfei, Zhang Lei, Semi-supervised left atrium segmentation with mutual consistency training, in: Medical image computing and computer assisted intervention–MICCAI 2021: 24th international conference, Strasbourg, France, September 27–October 1, 2021, proceedings, part II 24, Springer, 2021, pp. 297–306.

[54]

Xie Xiwang, Pan Xipeng, Zhang Weidong, An Jubai, A context hierarchical integrated network for medical image segmentation, Computers & Electrical Engineering 101 (2022).

[55]

Xu Ju, Li Mengzhang, Zhu Zhanxing, Automatic data augmentation for 3D medical image segmentation, in: Medical image computing and computer assisted intervention–MICCAI 2020: 23rd international conference, Lima, Peru, October 4–8, 2020, proceedings, part I 23, Springer, 2020, pp. 378–387.

[56]

Xu Xuanang, Sanford Thomas, Turkbey Baris, Xu Sheng, Wood Bradford J., Yan Pingkun, Shadow-consistent semi-supervised learning for prostate ultrasound segmentation, IEEE Transactions on Medical Imaging 41 (6) (2021) 1331–1345.

[57]

Xu Zihang, Xu Zhenghua, Zhang Shuo, Lukasiewicz Thomas, PCA: Semi-supervised segmentation with patch confidence adversarial training, 2022, arXiv preprint arXiv:2207.11683.

[58]

Yang Dong, Roth Holger, Xu Ziyue, Milletari Fausto, Zhang Ling, Xu Daguang, Searching learning strategy with reinforcement learning for 3D medical image segmentation, in: Medical image computing and computer assisted intervention–MICCAI 2019: 22nd international conference, Shenzhen, China, October 13–17, 2019, proceedings, part II 22, Springer, 2019, pp. 3–11.

[59]

Yap Moi Hoon, Goyal Manu, Osman Fatima, Martí Robert, Denton Erika, Juette Arne, et al., Breast ultrasound region of interest detection and lesion localisation, Artificial Intelligence in Medicine 107 (2020).

[60]

Yap Moi Hoon, Pons Gerard, Marti Joan, Ganau Sergi, Sentis Melcior, Zwiggelaar Reyer, et al., Automated breast ultrasound lesions detection using convolutional neural networks, IEEE Journal of Biomedical and Health Informatics 22 (4) (2017) 1218–1226.

[61]

Yelleni Sai Harsha, Kumari Deepshikha, Srijith P.K., et al., Monte Carlo DropBlock for modeling uncertainty in object detection, Pattern Recognition 146 (2024).

[62]

Yu Lequan, Wang Shujun, Li Xiaomeng, Fu Chi-Wing, Heng Pheng-Ann, Uncertainty-aware self-ensembling model for semi-supervised 3D left atrium segmentation, in: Medical image computing and computer assisted intervention–mICCAI 2019: 22nd international conference, Shenzhen, China, October 13–17, 2019, proceedings, part II 22, Springer, 2019, pp. 605–613.

[63]

Yu Xin, Yang Qi, Zhou Yinchi, Cai Leon Y, Gao Riqiang, Lee Ho Hin, et al., UNesT: local spatial representation learning with hierarchical transformer for efficient medical segmentation, Medical Image Analysis (2023).

[64]

Yun, Sangdoo, Han, Dongyoon, Oh, Seong Joon, Chun, Sanghyuk, Choe, Junsuk, & Yoo, Youngjoon (2019). Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 6023–6032).

[65]

Zare Mohsen, Ghasemi Mojtaba, Zahedi Amir, Golalipour Keyvan, Mohammadi Soleiman Kadkhoda, Mirjalili Seyedali, et al., A global best-guided firefly algorithm for engineering problems, Journal of Bionic Engineering (2023) 1–30.

[66]

Zhai Donghai, Hu Bijie, Gong Xun, Zou Haipeng, Luo Jun, ASS-GAN: Asymmetric semi-supervised GAN for breast ultrasound image segmentation, Neurocomputing 493 (2022) 204–216.

Digital Library

[67]

Zhang, Hongyi, Cisse, Moustapha, Dauphin, Yann N., & Lopez-Paz, David (2018). mixup: Beyond Empirical Risk Minimization. In International conference on learning representations.

[68]

Zhang Yichi, Jiao Rushi, Liao Qingcheng, Li Dongyang, Zhang Jicong, Uncertainty-guided mutual consistency learning for semi-supervised medical image segmentation, Artificial Intelligence in Medicine 138 (2023).

[69]

Zhao, Amy, Balakrishnan, Guha, Durand, Fredo, Guttag, John V., & Dalca, Adrian V. (2019). Data augmentation using learned transformations for one-shot medical image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 8543–8553).

[70]

Zhao Libo, Qian Xiaolong, Guo Yinghui, Song Jiaqi, Hou Jinbao, Gong Jun, MSKD: Structured knowledge distillation for efficient medical image segmentation, Computers in Biology and Medicine (2023).

[71]

Zheng Han, Lin Lanfen, Hu Hongjie, Zhang Qiaowei, Chen Qingqing, Iwamoto Yutaro, et al., Semi-supervised segmentation of liver using adversarial learning with deep atlas prior, in: Medical image computing and computer assisted intervention–MICCAI 2019: 22nd international conference, Shenzhen, China, October 13–17, 2019, proceedings, part VI 22, Springer, 2019, pp. 148–156.

[72]

Zhou Tianfei, Li Liulei, Bredell Gustav, Li Jianwu, Unkelbach Jan, Konukoglu Ender, Volumetric memory network for interactive medical image segmentation, Medical Image Analysis 83 (2023).

[73]

Zhou Wen-Yang, Yang Guo-Wei, Hu Shi-Min, Jittor-GAN: A fast-training generative adversarial network model zoo based on Jittor, Computational Visual Media 7 (2021) 153–157.

Recommendations

FDCT: Fusion-Guided Dual-View Consistency Training for semi-supervised tissue segmentation on MRI
Abstract
Accurate tissue segmentation on MRI is important for physicians to make diagnosis and treatment for patients. However, most of the models are only designed for single-task tissue segmentation, and tend to lack generality to other MRI tissue ...
Highlights
- We propose a universal semi-supervised tissue segmentation framework on MRI called FDCT.
- We improve the CBAM based on the task of this paper, which brings a boost to the network performance.
- We propose the SBOM that generates ...
Boosting Medical Image Segmentation with Partial Class Supervision
Biometric Recognition
Abstract
Medical image data are often limited due to expensive acquisition and annotation processes. Directly using such limited annotated samples can easily lead to the deep learning models overfitting on the training dataset. An alternative way is to ...
Poisson-based image editing for semi-supervised vitiligo lesion segmentation with limited annotations
Abstract
Vitiligo lesion segmentation is crucial for the assessment and treatment of vitiligo. There are two significant challenges in this problem, namely, the availability of dense segmentation annotations and the collection of large amounts of vitiligo ...
Highlights
- Newly proposed Bimodal Vitiligo Lesions Segmentation (BVLS) dataset for research in this area.
- A novel data augmentation strategy synthesizes high-quality samples from limited annotated data.
- Mining reliable pseudo-labels for ...

Comments

Information & Contributors

Information

Published In

cover image Neural Networks

Neural Networks Volume 177, Issue C

Sep 2024

298 pages

Issue’s Table of Contents

Elsevier Ltd.

Publisher

Elsevier Science Ltd.

United Kingdom

Publication History

Published: 24 July 2024

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

0
Total Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents