Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content

Advertisement

Springer Nature Link
Log in

Multi-category domain-dependent feature-based medical image translation

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

The challenge of inaccurate information containment in synthetic images during the process of cross-domain medical image translation could be resolved by using a common strategy of integrating the loss of the feature consistency of the real/synthetic image as a penalty factor into the loss function of the translator. However, the existing methods are capable of using only the “domain-independent” feature of the image when the aligned images are scarcity, which results in the under-utilization of the image information. In the present study, a novel feature consistency loss computing and integration method based on the “domain-dependent” features was proposed, and a multi-category feature consistency–cross-domain image translation (MFC-CIT) model was constructed. The present study is the first to utilize the image feature information related to image domain in the process of cross-domain medical image translation. In the proposed method, the MFC module was first trained on the basis of supervised learning on a limited number of paired real images. Next, cross-domain image translation training based on unsupervised learning was performed on unpaired datasets by the CIT module, and this process was constrained by the loss of feature consistency of the real/synthetic image obtained in the MFC module. The experimental results on two datasets demonstrate that the proposed method effectively improves the translation accuracy of synthetic images.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Bai, J., Chen, R., Liu, M.: Feature-attention module for context-aware image-to-image translation. Vis. Comput. 36, 2145–2159 (2020)

    Article  Google Scholar 

  2. Liang, X., Chen, L., Nguyen, D., Zhou, Z., Gu, X., Yang, M., Wang, J., Jiang, S.: Generating synthesized computed tomography (CT) from cone-beam computed tomography (CBCT) using CycleGAN for adaptive radiation therapy. Phys. Med. Biol. 64, 125002 (2019)

    Article  Google Scholar 

  3. Brou Boni, K.N.D., Klein, J., Vanquin, L., Wagner, A., Lacornerie, T., Pasquier, D., Reynaert, N.: MR to CT synthesis with multicenter data in the pelvic area using a conditional generative adversarial network. Phys. Med. Biol. 65, 075002 (2020)

    Article  Google Scholar 

  4. Nie, D., Trullo, R., Lian, J., Wang, L., Petitjean, C., Ruan, S., Wang, Q., Shen, D.: Medical image synthesis with deep convolutional adversarial networks. IEEE Trans. Biomed. Eng. 65, 2720–2730 (2018)

    Article  Google Scholar 

  5. Jiao, J., Namburete, A.I.L., Papageorghiou, A.T., Noble, J.A.: Anatomy-aware self-supervised fetal MRI synthesis from unpaired ultrasound images. In: Machine Learning in Medical Imaging (MLMI 2019). Lecture Notes in Computer Science, vol. 11861. (2019)

  6. Jiang, G., Lu, Y., Wei, J., Xu, Y.: Synthesize mammogram from digital breast tomosynthesis with gradient guided cGANs. In: Medical Image Computing and Computer Assisted Intervention (MICCAI 2019). Lecture Notes in Computer Science, vol. 11769, pp. 801–809. (2019)

  7. Chartsias, A., Joyce, T., Giuffrida, M., Tsaftaris, S.: Multimodal MR synthesis via modality-invariant latent representation. IEEE Trans. Med. Imaging 37, 803–814 (2018)

    Article  Google Scholar 

  8. Lartaud, P.-J., Dupont, C., Hallé, D., et al.: A conventional-to-spectral CT image translation augmentation workflow for robust contrast injection-independent organ segmentation. Med. Phys. 49(2), 1108–1122 (2022)

    Article  Google Scholar 

  9. Zhao, F. et al.: Harmonization of infant cortical thickness using surface-to-surface cycle-consistent adversarial networks. In: Medical Image Computing and Computer Assisted Intervention (MICCAI 2019). Lecture Notes in Computer Science, vol. 11767, pp. 475–483. (2019)

  10. Bentaieb, A., Hamarneh, G.: Adversarial stain transfer for histopathology image analysis. IEEE Trans. Med. Imaging 37, 792–802 (2018)

    Article  Google Scholar 

  11. Roy, S., Butman, J.A., Pham, D.L.: Synthesizing CT from ultrashort echo-time MR images via convolutional neural networks. In: Simulation and Synthesis in Medical Imaging (SASHIMI 2017). Lecture Notes in Computer Science, vol. 10557, pp. 24–32. (2017)

  12. Xiang, L., Wang, Q., Nie, D., Zhang, L., Jin, X., Qiao, Y., Shen, D.: Deep embedding convolutional neural network for synthesizing CT image from T1-weighted MR image. Med. Image Anal. 47, 31–44 (2018)

    Article  Google Scholar 

  13. Lei, Y., et al.: CBCT-based synthetic MRI generation for CBCT-Guided adaptive radiotherapy. In: Artificial Intelligence in Radiation Therapy (AIRT 2019) Lecture Notes in Computer Science, vol. 11850, pp. 154–161. (2019)

  14. Isola, P., Zhu, J., Zhou, T., Efros, A.: Image-to-image translation with conditional adversarial networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Honolulu. HI, vol. 632, pp. 5967–5976. (2017)

  15. Yurt, M., Dar, S.U.H., Erdem, A., Erdem, E., Oguz, K.K., Çukur, T.: MustGAN: multi-stream generative adversarial networks for MR image synthesis. Med Image Anal 70, 101944 (2021)

    Article  Google Scholar 

  16. Wang, M., Li, P.: A review of deformation models in medical image registration. J. Med. Biol. Eng. 39, 1–17 (2019)

    Article  Google Scholar 

  17. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Proceedings of International Conference on Neural Information Processing Systems Montréal, Canada, pp. 2672–2680. (2014)

  18. Zhu, J., Park, T., Isola, P., Efros, A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: 2017 IEEE International Conference on Computer Vision (ICCV), Venice, pp. 2242–2251. (2017)

  19. Liu, M., Thomas, B., Jan, K.: Unsupervised image-to-image translation networks. In: Annual Conference on Neural Information Processing Systems, Long Beach, USA, pp. 701–709. (2017)

  20. Joyce, T., Kozerke, S.: 3D medical image synthesis by factorised representation and deformable model learning. In: Simulation and Synthesis in Medical Imaging (SASHIMI 2019). Lecture Notes in Computer Science, vol. 11827. (2019)

  21. Wolterink, J.M., Dinkla, A.M., Savenije, M.H.F., Seevinck, P.R., van den Berg, C.A.T., Išgum, I.: Deep MR to CT synthesis using unpaired data. In: Simulation and Synthesis in Medical Imaging (SASHIMI 2017). Lecture Notes in Computer Science, vol. 10557. (2017)

  22. Welander, P., Karlsson, S., Eklund, A.: Generative adversarial networks for image-to-image translation on multi-contrast mr images-a comparison of cyclegan and unit (2018). https://arxiv.org/abs/1806.07777

  23. Cai, J., Zhang, Z., Cui, L., Zheng, Y., Yang, L.: Toward cross-modal organ translation and segmentation: a cycle-and shape-consistent generative adversarial network. Med. Image Anal. 52, 174–184 (2019)

    Article  Google Scholar 

  24. Yang, H., Sun, J., Carass, A., Zhao, C., Lee, J., Xu, Z., Prince, J.: Unpaired brain MR-to-CT synthesis using a structure-constrained CycleGAN. In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. (ML-CDS 2018). Lecture Notes in Computer Science, vol. 11045, pp. 174–182. (2018)

  25. Hiasa, Y., et al.: Cross-modality image synthesis from unpaired data using cyclegan: effects of gradient consistency loss and training data size. In: Simulation and Synthesis in Medical Imaging (SASHIMI 2019). Lecture Notes in Computer Science, vol. 11037, pp. 31–41. (2018)

  26. Wang, C., Papanastasiou, G., Tsaftaris, S., Yang, G., Gray, C., Newby, D., Macnaught, G., MacGillivray, T.: TPSDicyc: Improved deformation invariant cross-domain medical image synthesis. In: Machine Learning for Medical Image Reconstruction (MLMIR 2019). Lecture Notes in Computer Science, vol. 11905, pp. 245–254. (2019)

  27. Guo, X., Liu, J., Yuan, Y., et al.: Semantic-oriented labeled-to-unlabeled distribution translation for image segmentation. IEEE Trans. Med. Imaging 41(2), 434–445 (2022)

    Article  Google Scholar 

  28. Guo, L., et al.: CEUS-based classification of liver tumors with deep canonical correlation analysis and multi-kernel learning. In: 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Seogwipo, pp. 1748–1751. (2017)

  29. Hussein, S., Kandel, P., Corral, J.E., Bolan, W.C., Wallace, M.B., Bagci, U.: Deep multi-modal classification of intraductal papillary mucinous neoplasms (IPMN) with canonical correlation analysis. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), vol. 8363693, pp. 800–804 (2018)

  30. An L., et al.: A multi-level canonical correlation analysis scheme for standard-dose pet image estimation. In: Patch-Based Techniques in Medical Imaging (Patch-MI 2015). Lecture Notes in Computer Science, vol. 9467, pp. 1–9 (2015)

  31. Xing, F., Bennett, T., Ghosh, D.: Adversarial domain adaptation and pseudo-labeling for cross-modality microscopy image quantification. In: Medical Image Computing and Computer Assisted Intervention (MICCAI 2019). Lecture Notes in Computer Science, vol. 11764, pp. 740–749 (2019)

  32. Sun, S.: A survey of multi-view machine learning. Neural Comput. Appl. 23, 2031–2038 (2013)

    Article  Google Scholar 

  33. Hardoon, D., Szedmak, S., Shawe-Taylor, J.: Canonical correlation analysis: an overview with application to learning methods. Neural Comput. 16, 2639–2664 (2004)

    Article  Google Scholar 

  34. Karageorgiou, E., Lewis, S.M., McCarten, J.R., Leuthold, A.C., Hemmy, L.S., McPherson, S.E., Rottunda, S.J., Rubins, D.M., Georgopoulos, A.P.: Canonical correlation analysis of synchronous neural interactions and cognitive deficits in Alzheimer’s dementia. Phys. Med. Biol. 9, 056003 (2012)

    Google Scholar 

  35. Andrew, G., Arora, R., Bilmes, J., Livescu, K.: Deep canonical correlation analysis. In: Proceedings of the 30th International Conference on Machine Learning, pp. 1247–1255 (2013)

  36. Engelen, A., Niessen, W.J., Klein, S., Groen, H.C., Verhagen, H.J., Wentzel, J.J., van der Lugt, A., de Bruijne, M.: Multi-feature-based plaque characterization in ex vivo MRI trained by registration to 3D histology. Phys. Med. Biol. 57, 241–256 (2012)

    Article  Google Scholar 

  37. Heusel, M., et al.: Gans trained by a two time-scale update rule converge to a nash equilibrium. CoRR abs/1706.08500 (2017). http://arxiv.org/abs/1706.08500

  38. Sheikh, H.R., et al.: An information fidelity criterion for image quality assessment using natural scene statistics. IEEE Trans. Image Process. 14(12), 2117–2128 (2005)

    Article  Google Scholar 

  39. Szegedy, C., et al.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826. (2016)

  40. Zhang, Y., Liu, M., Yu, F., Zeng, T., Wang, Y.: An O-shape neural network with attention modules to detect junctions in biomedical images without segmentation. IEEE J. Biomed. Health Inform. 26(2), 774–785 (2021)

    Article  Google Scholar 

  41. Chen, Y., Lin, Y., Ding, J., Li, C., Zeng, Y., Xie, W., Huang, J.: Multi-domain medical image translation generation for lung image classification based on generative adversarial networks. Comput. Methods Prog. Biomed. (2022)

  42. Yu, F., Liu, M., Chen, W., Wen, H., Wang, Y., Zeng, T.: Automatic repair of 3-D neuron reconstruction based on topological feature points and an MOST-based repairer. IEEE Trans. Instrum. Meas. 70, 1–13 (2020)

    Google Scholar 

  43. Liu, J., Wen, X., Nie, W., Su, Y., Jing, P., Yang, X.: Residual-guided multiscale fusion network for bit-depth enhancement. Trans Circuits Syst Video Technol 32(5), 2773–2786 (2022)

    Article  Google Scholar 

  44. Armanious, K., Jiang, C., Fischer, M., et al.: MedGAN: medical image translation using GANs. Comput. Med. Imaging Graph. 79(2020), 101684 (2020)

    Article  Google Scholar 

  45. Wang, L., Sun, D., Yuan, Z., et al.: Multi-view clustering based on graph learning and view diversity learning. Vis. Comput. (2022)

  46. Karthik, K., Kamath, S.S.: A deep neural network model for content-based medical image retrieval with multi-view classification. Vis. Comput. 37, 1837–1850 (2021)

    Article  Google Scholar 

  47. Bayoudh, K., Knani, R., Hamdaoui, F., et al.: A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets. Vis. Comput. (2021)

  48. Ninon, B., et al.: Iterative framework for the joint segmentation and CT synthesis of MR images: application to MRI-only radiotherapy treatment planning. Phys. Med. Biol. 62, 4237–4253 (2017)

    Article  Google Scholar 

  49. Liu, G., Han, J.: Cross-domain object detection using minimized instance shift image–image translation. Vis. Comput. (2022)

Download references

Acknowledgements

The research would like to thank the financial supports of the Key Project for University of Department of Education of Guangdong Province of China Funds (Natural) under Grant 2019GZDXM005.

Author information

Authors and Affiliations

  1. Guangdong Polytechnic of Science and Technology, Zhuhai, Guangdong, China

    Ning Lu

  2. College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, China

    Yizhou Chen

Authors
  1. Ning Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yizhou Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yizhou Chen.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix: The structure adopted by the deep network in this paper

Appendix: The structure adopted by the deep network in this paper

The structure of the generator and discriminator in this paper is shown in columns 2 and 3 of Table

Table 6 The structure adopted by the deep network in this paper, where “kernel” represents the convolution kernel, and “size” is the size of the convolution kernel
Full size table

6, respectively. The CNN structure of \(f_{{\text{var}}}\) used by MFC in this paper is shown in column 4 of Table 6.

Here, InstanceNorm2d is a batch normalization operation, max-pooling is a pooling operation, and ReLU, LeakyReLU (0.2), and sigmoid are activation functions. The “ × 9 layers” in layer 4 of the second column (generator) represent that nine identical network layers are connected.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lu, N., Chen, Y. Multi-category domain-dependent feature-based medical image translation. Vis Comput 40, 4519–4538 (2024). https://doi.org/10.1007/s00371-023-03096-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-023-03096-2

Keywords

Search

Navigation