Abstract
Deep neural networks have recently been succeessful in the field of medical image segmentation; however, they are typically subject to performance degradation problems when well-trained models are tested in another new domain with different data distributions. Given that annotated cross-domain images may inaccessible, unsupervised domain adaptation methods that transfer learnable information from annotated source domains to unannotated target domains with different distributions have attracted substantial attention. Many methods leverage image-level or pixel-level translation networks to align domain-invariant information and mitigate domain shift issues. However, These methods rarely perform well when there is a large domain gap. A new unsupervised deep consistency learning adaptation network, which adopts input space consistency learning and output space consistency learning to realize unsupervised domain adaptation and cardiac structural segmentation, is introduced in this paper The framework mainly includes a domain translation path and a cross-modality segmentation path. In domain translation path, a symmetric alignment generator network with attention to cross-modality features and anatomy is introduced to align bidirectional domain features. In the segmentation path, entropy map minimization, output probability map minimization and segmentation prediction minimization are leveraged to align the output space features. The model conducts supervised learning to extract source domain features and conducts unsupervised deep consistency learning to extract target domain features. Through experimental testing on two challenging cross-modality segmentation tasks, our method has robust performance compared to that of previous methods. Furthermore, ablation experiments are conducted to confirm the effectiveness of our framework.
Graphical abstract
Similar content being viewed by others
Data Availability
Data related to the current study are available from the corresponding author on reasonable request.
Code Availability
The codes used during the study are available from the First author by request.
References
Bateson M, Dolz J, Kervadec H, Lombaert H, Ayed IB (2021) Constrained domain adaptation for image segmentation. IEEE Trans Med Imaging 40(7):1875–1887
Bernard O, Lalande A, Zotti C, Cervenansky F, Yang X, Heng PA, Cetin I, Lekadir K, Camara O, Ballester MAG et al (2018) Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: is the problem solved? IEEE Trans Med Imaging 37(11):2514–2525
Borgwardt KM, Gretton A, Rasch MJ, Kriegel HP, Schölkopf B, Smola AJ (2006) Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics 22(14):e49–e57
Bousmalis K, Trigeorgis G, Silberman N, Krishnan D, Erhan D (2016) Domain separation networks. Advances in neural information processing systems 29
Chen C, Chen Z, Jiang B, Jin X (2019) Joint domain alignment and discriminative feature learning for unsupervised deep domain adaptation. Proc AAAI Conf Artif Intell 33:3296–3303
Chen C, Dou Q, Chen H, Qin J, Heng PA (2019) Synergistic image and feature adaptation: Towards cross-modality domain adaptation for medical image segmentation. Proc AAAI Conf Artif Intell 33:865–872
Chen C, Dou Q, Chen H, Qin J, Heng PA (2020) Unsupervised bidirectional cross-modality adaptation via deeply synergistic image and feature alignment for medical image segmentation. IEEE Trans Med Imaging 39(7):2494–2505
Chen X, Lian C, Wang L, Deng H, Kuang T, Fung SH, Gateno J, Shen D, Xia JJ, Yap PT (2021) Diverse data augmentation for learning image segmentation with cross-modality annotations. Med Image Anal 71:102060
Chen X, Kuang T, Deng H, Fung SH, Gateno J, Xia JJ, Yap PT (2022) Dual adversarial attention mechanism for unsupervised domain adaptive medical image segmentation. IEEE Transactions on Medical Imaging
Cordts M, Omran M, Ramos S, Rehfeld T, Enzweiler M, Benenson R, Franke U, Roth S, Schiele B (2016) The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 3213–3223
Csurka G, Volpi R, Chidlovskii B (2021) Unsupervised domain adaptation for semantic image segmentation: a comprehensive survey. arXiv:2112.03241
Cui H, Yuwen C, Jiang L, Xia Y, Zhang Y (2021) Bidirectional cross-modality unsupervised domain adaptation using generative adversarial networks for cardiac image segmentation. Comput Biol Med 136:104726
Cui Z, Li C, Du Z, Chen N, Wei G, Chen R, Yang L, Shen D, Wang W (2021) Structure-driven unsupervised domain adaptation for cross-modality cardiac segmentation. IEEE Trans Med Imaging 40(12):3604–3616
Dou Q, Ouyang C, Chen C, Chen H, Glocker B, Zhuang X, Heng PA (2019) PnP-adaNet: Plug-and-play adversarial domain adaptation network at unpaired cross-modality cardiac segmentation. IEEE Access 7:99065–99076
Dou Q, Liu Q, Heng PA, Glocker B (2020) Unpaired multi-modal segmentation via knowledge distillation. IEEE Trans Med Imaging 39(7):2415–2425
Ge Y, Xue Z, Cao T, Liao S (2019) Unpaired whole-body MR to CT synthesis with correlation coefficient constrained adversarial learning. In: Medical imaging 2019: Image Processing, SPIE, vol 10949. pp 28–35
Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. Advances in neural information processing systems 27
Han X, Qi L, Yu Q, Zhou Z, Zheng Y, Shi Y, Gao Y (2021) Deep symmetric adaptation network for cross-modality medical image segmentation. IEEE Trans Med Imaging 41(1):121–132
Hoffman J, Tzeng E, Park T, Zhu JY, Isola P, Saenko K, Efros A, Darrell T (2018) Cycada: Cycle-consistent adversarial domain adaptation. In: International conference on machine learning, Pmlr. pp 1989–1998
Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 1125–1134
Jafari M, Francis S, Garibaldi JM, Chen X (2022) LMISA: A lightweight multi-modality image segmentation network via domain adaptation using gradient magnitude and shape constraint. Medical Image Analysis p 102536
Kim T, Cha M, Kim H, Lee JK, Kim J (2017) Learning to discover cross-domain relations with generative adversarial networks. In: International conference on machine learning, PMLR. pp 1857–1865
Kuroki S, Charoenphakdee N, Bao H, Honda J, Sato I, Sugiyama M (2019) Unsupervised domain adaptation based on source-guided discrepancy. Proc AAAI Conf Artif Intell 33:4122–4129
Lee AH, Hiramatsu N (2011) Role of n-3 series polyunsaturated fatty acids in cardiovascular disease prevention. Nutr Diet Suppl 3:93
Li K, Yu L, Wang S, Heng PA (2020) Towards cross-modality medical image segmentation with online mutual knowledge distillation. Proc AAAI Conf Artif Intell 34:775–783
Liao X, Qian Y, Chen Y, Xiong X, Wang Q, Heng PA (2020) Mmtlnet: Multi-modality transfer learning network with adversarial training for 3D whole heart segmentation. Comput Med Imaging Graph 85:101785
Long M, Cao Y, Wang J, Jordan M (2015) Learning transferable features with deep adaptation networks. In: International conference on machine learning, PMLR. pp 97–105
Long M, Zhu H, Wang J, Jordan MI (2016) Unsupervised domain adaptation with residual transfer networks. Advances in neural information processing systems 29
Lu C, Zheng S, Gupta G (2022) Unsupervised domain adaptation for cardiac segmentation: Towards structure mutual information maximization. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp 2588–2597
Van der Maaten L, Hinton G (2008) Visualizing data using t-SNE. J Mach Learn Res 9(11)
Mancini M, Porzi L, Bulo SR, Caputo B, Ricci E (2018) Boosting domain adaptation by discovering latent domains. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 3771–3780
Mollet NR, Dymarkowski S, Bogaert J (2003) MRI and CT revealing carcinoid heart disease. Eur Radiol 13(6):L14–L18
Park T, Liu MY, Wang TC, Zhu JY (2019) Semantic image synthesis with spatially-adaptive normalization. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp 2337–2346
Pei C, Wu F, Huang L, Zhuang X (2021) Disentangle domain features for cross-modality cardiac image segmentation. Med Image Anal 71:102078
Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention, Springer. pp 234–241
Shen D, Wu G, Suk HI (2017) Deep learning in medical image analysis. Annu Rev Biomed Eng 19:221
Sun B, Saenko K (2016) Deep coral: Correlation alignment for deep domain adaptation. In: European conference on computer vision, Springer. pp 443–450
Tomar D, Lortkipanidze M, Vray G, Bozorgtabar B, Thiran JP (2021) Self-attentive spatial adaptive normalization for cross-modality domain adaptation. IEEE Trans Med Imaging 40(10):2926–2938
Tsai YH, Hung WC, Schulter S, Sohn K, Yang MH, Chandraker M (2018) Learning to adapt structured output space for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 7472–7481
Tzeng E, Hoffman J, Saenko K, Darrell T (2017) Adversarial discriminative domain adaptation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 7167–7176
Vesal S, Gu M, Kosti R, Maier A, Ravikumar N (2021) Adapt everywhere: unsupervised adaptation of point-clouds and entropy minimization for multi-modal cardiac image segmentation. IEEE Trans Med Imaging 40(7):1838–1851
Vs V, Gupta V, Oza P, Sindagi VA, Patel VM (2021) MeGA-CDA: Memory guided attention for category-aware unsupervised domain adaptive object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp 4516–4526
Vu TH, Jain H, Bucher M, Cord M, Pérez P (2019) Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp 2517–2526
Wang M, Deng W (2018) Deep visual domain adaptation: A survey. Neurocomputing 312:135–153
Wang Q, Du Y, Fan H, Ma C (2022) Towards collaborative appearance and semantic adaptation for medical image segmentation. Neurocomputing 491:633–643
Wang R, Zheng G (2022) CyCMIS: Cycle-consistent cross-domain medical image segmentation via diverse image augmentation. Med Image Anal 76:102328
Wang S, Yu L, Li K, Yang X, Fu CW, Heng PA (2019) Boundary and entropy-driven adversarial learning for fundus image segmentation. In: International conference on medical image computing and computer-assisted intervention, Springer. pp 102–110
Wilson G, Cook DJ (2020) A survey of unsupervised deep domain adaptation. ACM Trans Intell Syst Technol (TIST) 11(5):1–46
Wu F, Zhuang X (2020) CF distance: A new domain discrepancy metric and application to explicit domain adaptation for cross-modality cardiac image segmentation. IEEE Trans Med Imaging 39(12):4274–4285
Wu F, Zhuang X (2021) Unsupervised domain adaptation with variational approximation for cardiac segmentation. IEEE Trans Med Imaging 40(12):3555–3567
Yi Z, Zhang H, Tan P, Gong M (2017) Dualgan: Unsupervised dual learning for image-to-image translation. In: Proceedings of the IEEE international conference on computer vision. pp 2849–2857
Zhou Z, Qi L, Yang X, Ni D, Shi Y (2022) Generalizable cross-modality medical image segmentation via style augmentation and dual normalization. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp 20856–20865
Zhu JY, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision. pp 2223–2232
Zhuang X (2018) Multivariate mixture model for myocardial segmentation combining multi-source images. IEEE Trans Pattern Anal Mach Intell 41(12):2933–2946
Zhuang X, Shen J (2016) Multi-scale patch and multi-modality atlases for whole heart segmentation of MRI. Med Image Anal 31:77–87 t
Zhuang X, Li L, Payer C, Štern D, Urschler M, Heinrich MP, Oster J, Wang C, Smedby Ö, Bian C et al (2019) Evaluation of algorithms for multi-modality whole heart segmentation: an open-access grand challenge. Med Image Anal 58:101537
Zhuang X, Xu J, Luo X, Chen C, Ouyang C, Rueckert D, Campello VM, Lekadir K, Vesal S, RaviKumar N, et al. (2020) Cardiac segmentation on late gadolinium enhancement MRI: a benchmark study from multi-sequence cardiac MR segmentation challenge. arXiv:2006.12434
Zou Y, Yu Z, Kumar B, Wang J (2018) Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In: Proceedings of the European conference on computer vision (ECCV). pp 289–305
Funding
This work was supported in part by the National Natural Science Foundation of China (Grant No. 61976126), Shandong Nature Science Foundation of China (No. ZR2019MF003).
Author information
Authors and Affiliations
Contributions
Dapeng Li: Conceptualization, Methodology, Software, Writing the original draft. Yanjun Peng: Data curation, Supervision. Jindong Sun: Investigation, Formal analysis, Software. Yanfei Guo: Visualization, Writing-review and editing.
Corresponding author
Ethics declarations
Conflicts of interest
The authors declare there are no conflicts of interest regarding the publication of this paper.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Li, D., Peng, Y., Sun, J. et al. Unsupervised deep consistency learning adaptation network for cardiac cross-modality structural segmentation. Med Biol Eng Comput 61, 2713–2732 (2023). https://doi.org/10.1007/s11517-023-02833-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11517-023-02833-y