DBGAN: Dual Branch Generative Adversarial Network for Multi-Modal MRI Translation
Authors: 
Jun Lyu, Shouang Yan, and M. Shamim HossainAuthors Info & Claims
Jun Lyu, Shouang Yan, and M. Shamim HossainAuthors Info & ClaimsArticle No.: 235, Pages 1 - 22
Abstract
Existing magnetic resonance imaging translation models rely on generative adversarial networks, primarily employing simple convolutional neural networks. Unfortunately, these networks struggle to capture global representations and contextual relationships within magnetic resonance images. While the advent of Transformers enables capturing long-range feature dependencies, they often compromise the preservation of local feature details. To address these limitations and enhance both local and global representations, we introduce DBGAN, a novel dual-branch generative adversarial network. In this framework, the Transformer branch comprises sparse attention blocks and dense self-attention blocks, allowing for a wider receptive field while simultaneously capturing local and global information. The convolutional neural network branch, built with integrated residual convolutional layers, enhances local modeling capabilities. Additionally, we propose a fusion module that cleverly integrates features extracted from both branches. Extensive experimentation on two public datasets and one clinical dataset validates significant performance improvements with DBGAN. On Brats2018, it achieves a 10% improvement in MAE, 3.2% in PSNR, and 4.8% in SSIM for image generation tasks compared to RegGAN. Notably, the generated MRIs receive positive feedback from radiologists, underscoring the potential of our proposed method as a valuable tool in clinical settings.
References
[1]
Andy Adam, Adrian K. Dixon, Jonathan H. Gillard, Cornelia Schaefer-Prokop, Ronald G. Grainger, and David J. Allison. 2014. Grainger & Allison’s Diagnostic Radiology E-Book. Elsevier Health Sciences.
[2]
Deblina Bhattacharjee, Seungryong Kim, Guillaume Vizier, and Mathieu Salzmann. 2020. DUNIT: Detection-based unsupervised image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4787–4796.
[3]
Andrew Brock, Jeff Donahue, and Karen Simonyan. 2018. Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096 (2018).
[4]
Anirudh Chandrashekar, Ashok Handa, Natesh Shivakumar, Pierfrancesco Lapolla, Vicente Grau, and Regent Lee. 2020. A deep learning approach to generate contrast-enhanced computerised tomography angiography without the use of intravenous contrast agents. arXiv preprint arXiv:2003.01223 (2020).
[5]
Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L. Yuille, and Yuyin Zhou. 2021. TransUNet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306 (2021).
[6]
Runfa Chen, Wenbing Huang, Binghui Huang, Fuchun Sun, and Bin Fang. 2020. Reusing discriminators for encoding: Towards unsupervised image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8168–8177.
[7]
Nicolas Cordier, Hervé Delingette, Matthieu Lê, and Nicholas Ayache. 2016. Extended modality propagation: Image synthesis of pathological cases. IEEE Transactions on Medical Imaging 35, 12 (2016), 2598–2608.
[8]
Pedro Costa, Adrian Galdran, Maria Inês Meyer, Michael David Abramoff, Meindert Niemeijer, Ana Maria Mendonça, and Aurélio Campilho. 2017. Towards adversarial retinal image synthesis. arXiv preprint arXiv:1701.08974 (2017).
[9]
Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A. Bharath. 2018. Generative adversarial networks: An overview. IEEE Signal Processing Magazine 35, 1 (2018), 53–65.
[10]
Yin Dai, Yifan Gao, and Fayu Liu. 2021. TransMed: Transformers advance multi-modal medical image classification. Diagnostics 11, 8 (2021), 1384.
[11]
Onat Dalmaz, Mahmut Yurt, and Tolga Çukur. 2022. ResViT: Residual vision transformers for multimodal medical image synthesis. IEEE Transactions on Medical Imaging 41, 10 (2022), 2598–2614.
[12]
Salman U. H. Dar, Mahmut Yurt, Levent Karacan, Aykut Erdem, Erkut Erdem, and Tolga Cukur. 2019. Image synthesis in multi-contrast MRI with conditional generative adversarial networks. IEEE Transactions on Medical Imaging 38, 10 (2019), 2375–2388.
[13]
Rahatara Ferdousi, Nabila Mabruba, Fedwa Laamarti, Abdulmotaleb El Saddik, and Chunsheng Yang. 2022. Non-invasive anemia detection from conjunctival images. In Proceedings of the International Conference on Smart Multimedia. 189–201.
[14]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020. Generative adversarial networks. Communications of the ACM 63, 11 (2020), 139–144.
[15]
Alper Güngör, Baris Askin, Damla Alptekin Soydan, Emine Ulku Saritas, Can Barış Top, and Tolga Çukur. 2022. TranSMS: Transformers for super-resolution calibration in magnetic particle imaging. IEEE Transactions on Medical Imaging 41, 12 (2022), 3562–3574.
[16]
M. Shamim Hossain, Ghulam Muhammad, and Atif Al Amri. 2019. Smart healthcare monitoring: A voice pathology detection paradigm for smart cities. Multimedia Systems 25, 5 (2019), 565–575.
[17]
Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz. 2018. Multimodal unsupervised image-to-image translation. In Proceedings of the European Conference on Computer Vision. 172–189.
[18]
Yawen Huang, Ling Shao, and Alejandro F. Frangi. 2017. Simultaneous super-resolution and cross-modality synthesis of 3D medical images using weakly-supervised joint convolutional sparse coding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6070–6079.
[19]
Juan Eugenio Iglesias, Ender Konukoglu, Darko Zikic, Ben Glocker, Koen Van Leemput, and Bruce Fischl. 2013. Is synthesizing MRI contrast useful for inter-modality analysis? In Medical Image Computing and Computer-Assisted Intervention—MICCAI 2013. Lecture Notes in Computer Science, Vol. 8149. Springer, 631–638.
[20]
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. 2017. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1125–1134.
[21]
Yuanfeng Ji, Ruimao Zhang, Huijie Wang, Zhen Li, Lingyun Wu, Shaoting Zhang, and Ping Luo. 2021. Multi-compound transformer for accurate biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention—MICCAI 2021. Lecture Notes in Computer Science, Vol. 12901. Springer, 326–336.
[22]
Konstantinos Kamnitsas, Christian Ledig, Virginia F. J. Newcombe, Joanna P. Simpson, Andrew D. Kane, David K. Menon, Daniel Rueckert, and Ben Glocker. 2017. Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Medical Image Analysis 36 (2017), 61–78.
[23]
Hadi Kazemi, Sobhan Soleymani, Fariborz Taherkhani, Seyed Iranmanesh, and Nasser Nasrabadi. 2018. Unsupervised image-to-image translation using domain-specific variational information bound. Advances in Neural Information Processing Systems 31 (2018), 1–11.
[24]
Vasant Kearney, Benjamin P. Ziemer, Alan Perry, Tianqi Wang, Jason W. Chan, Lijun Ma, Olivier Morin, Sue S. Yom, and Timothy D. Solberg. 2020. Attention-aware discrimination for MR-to-CT image translation using cycle-consistent generative adversarial networks. Radiology: Artificial Intelligence 2, 2 (2020), e190027.
[25]
Soohyun Kim, Jongbeom Baek, Jihye Park, Gyeongnyeon Kim, and Seungryong Kim. 2022. InstaFormer: Instance-aware image-to-image translation with transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 18321–18331.
[26]
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[27]
Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira. 2017. On convergence and stability of GANs. arXiv preprint arXiv:1705.07215 (2017).
[28]
Lingke Kong, Chenyu Lian, Detian Huang, Zhenjiang Li, Yanle Hu, and Qichao Zhou. 2021. Breaking the dilemma of medical image-to-image translation. Advances in Neural Information Processing Systems 34 (2021), 1964–1978.
[29]
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin Transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 10012–10022.
[30]
Yanmei Luo, Yan Wang, Chen Zu, Bo Zhan, Xi Wu, Jiliu Zhou, Dinggang Shen, and Luping Zhou. 2021. 3D transformer-GAN for high-quality PET reconstruction. In Medical Image Computing and Computer-Assisted Intervention—MICCAI 2021. Lecture Notes in Computer Science, Vol. 12906. Springer, 276–285.
[31]
Bjoern H. Menze, Andras Jakab, Stefan Bauer, Jayashree Kalpathy-Cramer, Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, Levente Lanczi, Elizabeth Gerstner, Marc-Andre Weber, Tal Arbel, Brian B. Avants, Nicholas Ayache, Patricia Buendia, D. Louis Collins, Nicolas Cordier, Jason J. Corso, Antonio Criminisi, Tilak Das, Herve Delingette, Cagatay Demiralp, Christopher R. Durst, Michel Dojat, Senan Doyle, Joana Festa, Florence Forbes, Ezequiel Geremia, Ben Glocker, Polina Golland, Xiaotao Guo, Andac Hamamci, Khan M. Iftekharuddin, Raj Jena, Nigel M. John, Ender Konukoglu, Danial Lashkari, Jose Antonio Mariz, Raphael Meier, Sergio Pereira, Doina Precup. Stephen J. Price, Tammy Riklin Raviv, Syed M. S. Reza, Michael Ryan, Duygu Sarikaya, Lawrence Schwartz, Hoo-Chang Shin, Jamie Shotton, Carlos A. Silva, Nuno Sousa, Nagesh K. Subbanna, Gabor Szekely, Thomas J. Taylor, Owen M. Thomas, Nicholas J. Tustison, Gozde Unal, Flor Vasseur, Max Wintermark, Dong Hye Ye, Liang Zhao, Binsheng Zhao, Darko Zikic, Marcel Prastawa, Mauricio Reyes, and Koen Van Leemput. 2014. The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Transactions on Medical Imaging 34, 10 (2014), 1993–2024.
[32]
Tony C. W. Mok and Albert Chung. 2022. Affine medical image registration with Coarse-to-Fine Vision Transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 20835–20844.
[33]
Dong Nie, Roger Trullo, Jun Lian, Li Wang, Caroline Petitjean, Su Ruan, Qian Wang, and Dinggang Shen. 2018. Medical image synthesis with deep convolutional adversarial networks. IEEE Transactions on Biomedical Engineering 65, 12 (2018), 2720–2730.
[34]
Augustus Odena, Jacob Buckman, Catherine Olsson, Tom Brown, Christopher Olah, Colin Raffel, and Ian Goodfellow. 2018. Is generator conditioning causally related to GAN performance? In Proceedings of the International Conference on Machine Learning. 3849–3858.
[35]
Muzaffer Özbey, Onat Dalmaz, Salman U. H. Dar, Hasan A. Bedel, Şaban Özturk, Alper Güngör, and Tolga Çukur. 2023. Unsupervised medical image translation with adversarial diffusion models. IEEE Transactions on Medical Imaging 42, 12 (2023), 3524–3539.
[36]
Abdur Rahman, M. Shamim Hossain, Nabil A. Alrajeh, and Fawaz Alsolami. 2021. Adversarial examples—Security threats to COVID-19 deep learning systems in medical IoT devices. IEEE Internet of Things Journal 8, 12 (2021), 9603–9610.
[37]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015. Lecture Notes in Computer Science, Vol. 9351. Springer, 234–241.
[38]
Sara Sabour, Nicholas Frosst, and Geoffrey E. Hinton. 2017. Dynamic routing between capsules. Advances in Neural Information Processing Systems 30 (2017), 1–11.
[39]
Jeffrey Tsao. 2010. Ultrafast imaging: Principles, pitfalls, solutions, and applications. Journal of Magnetic Resonance Imaging 32, 2 (2010), 252–266.
[40]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems 30 (2017), 1–11.
[41]
J. E. Vranic, N. M. Cross, Y. Wang, D. S. Hippe, E. De Weerdt, and M. Mossa-Basha. 2019. Compressed sensing–sensitivity encoding (CS-SENSE) accelerated brain imaging: Reduced scan time without reduced image quality. American Journal of Neuroradiology 40, 1 (2019), 92–98.
[42]
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. 2018. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7794–7803.
[43]
Yan Wang, Yusen Li, Gang Wang, and Xiaoguang Liu. 2022. Multi-scale attention network for single image super-resolution. arXiv preprint arXiv:2209.14145 (2022).
[44]
Xuezhi Xiang, Kaixu Zhang, Yulong Qiao, and Abdulmotaleb El Saddik. 2023. EMHIFormer: An enhanced multi-hypothesis interaction transformer for 3D human pose estimation in video. Journal of Visual Communication and Image Representation 95 (2023), 103890.
[45]
Bingyu Xin, Yifan Hu, Yefeng Zheng, and Hongen Liao. 2020. Multi-modality generative adversarial networks with tumor consistency loss for brain MR image synthesis. In Proceedings of the 2020 IEEE 17th International Symposium on Biomedical Imaging.IEEE, 1803–1807.
[46]
Shouang Yan, Chengyan Wang, Weibo Chen, and Jun Lyu. 2022. Swin Transformer-based GAN for multi-modal medical image translation. Frontiers in Oncology 12 (2022), 942511.
[47]
Jure Zbontar, Florian Knoll, Anuroop Sriram, Tullie Murrell, Zhengnan Huang, Matthew J. Muckley, Aaron Defazio, Ruben Stern, Patricia Johnson, Mary Bruno, Marc Parente, Krzysztof J. Geras, Joe Katsnelson, Hersh Chandarana, Zizhao Zhang, Michal Drozdzal, Adriana Romero, Michael Rabbat, Pascal Vincent, Nafissa Yakubova, James Pinkerton, Duo Wang, Erich Owens, C. Lawrence Zitnick, Michael P. Recht, Daniel K. Sodickson, and Yvonne W. Lui. 2018. fastMRI: An open dataset and benchmarks for accelerated MRI. arXiv preprint arXiv:1811.08839 (2018).
[48]
Bo Zhan, Di Li, Xi Wu, Jiliu Zhou, and Yan Wang. 2021. Multi-modal MRI image synthesis via GAN with multi-scale gate mergence. IEEE Journal of Biomedical and Health Informatics 26, 1 (2021), 17–26.
[49]
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. 2019. Self-attention generative adversarial networks. In Proceedings of the International Conference on Machine Learning. 7354–7363.
[50]
Jiale Zhang, Yulun Zhang, Jinjin Gu, Yongbing Zhang, Linghe Kong, and Xin Yuan. 2022. Accurate image restoration with attention retractable transformer. arXiv preprint arXiv:2210.01427 (2022).
[51]
Kai Zhang, Yawei Li, Jingyun Liang, Jiezhang Cao, Yulun Zhang, Hao Tang, Radu Timofte, and Luc Van Gool. 2022. Practical blind denoising via Swin-Conv-UNet and data synthesis. arXiv preprint arXiv:2203.13278 (2022).
[52]
Rui Zhang, Tomas Pfister, and Jia Li. 2019. Harmonic unpaired image-to-image translation. arXiv preprint arXiv:1902.09727 (2019).
[53]
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision. 2223–2232.
Index Terms
- DBGAN: Dual Branch Generative Adversarial Network for Multi-Modal MRI Translation
Recommendations
Mask-guided generative adversarial network for MRI-based CT synthesis
AbstractSynthetic computed tomography (sCT) images from magnetic resonance imaging (MRI) data have broad applications in clinical medicine, including radiation oncology and surgical planning. With the development of deep learning technology in medical ...
Multi-loss Super-Resolution Generative Adversarial Network
AIPR '23: Proceedings of the 2023 6th International Conference on Artificial Intelligence and Pattern RecognitionSingle-image super-resolution (SISR) based on deep neural networks has achieved excellent performance in recent years. However, how to recover texture details is still a challenging problem in the field of super-resolution. In this paper, in order to ...
Low-dose CT denoising using a Progressive Wasserstein generative adversarial network
AbstractLow-dose computed tomography (LDCT) imaging can greatly reduce the radiation dose imposed on the patient. However, image noise and visual artifacts are inevitable when the radiation dose is low, which has serious impact on the clinical medical ...
Highlights- Progressive Wasserstein generative adversarial network for low-dose computed tomography denoising
- Reduce the number of parameters with recursive computation
- Hybrid loss function for medical image denoising
Comments
Information & Contributors
Information
Published In
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 13 June 2024
Online AM: 10 April 2024
Accepted: 29 March 2024
Revised: 14 March 2024
Received: 23 January 2024
Published in TOMM Volume 20, Issue 8
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Researchers Supporting Project
- King Saud University, Riyadh, Saudi Arabia
- National Natural Science Foundation of China
- Yantai Basic Research Key Project
- Youth Innovation Science and Technology Support Program of Shandong Province
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 100Total Downloads
- Downloads (Last 12 months)100
- Downloads (Last 6 weeks)35
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in