Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article

Iterative Temporal-spatial Transformer-based Cardiac T1 Mapping MRI Reconstruction

Published: 08 March 2024 Publication History
  • Get Citation Alerts
  • Abstract

    The precise reconstruction of accelerated magnetic resonance imaging (MRI) brings about notable advantages, such as enhanced diagnostic precision and decreased examination costs. In contrast, traditional cardiac MRI necessitates repetitive acquisitions across multiple heartbeats, resulting in prolonged acquisition times. Significant strides have been made in accelerating MRI through deep learning-based reconstruction methods. However, these existing methods encounter certain limitations: (1) The intricate nature of heart reconstruction involving multiple complex time-series data poses a challenge in exploring nonlinear dependencies between temporal contexts. (2) Existing research often overlooks weight sharing in iterative frameworks, impeding the effective capturing of non-local information and, consequently, limiting improvements in model performance. In order to improve cardiac MRI reconstruction, we propose a novel temporal-spatial transformer with a strategy in this study. Based on the multi-level encoder and decoder transformer architecture, we conduct multi-level spatiotemporal information feature aggregation over several adjacent views, that create nonlinear dependencies among features and efficiently learn important information among adjacent cardiac temporal frames. Additionally, in order to improve contextual awareness between neighboring views, we add cross-view attention for temporal information fusion. Furthermore, we introduce an iterative strategy for training weights during the reconstruction process, which improves feature fusion in critical locations and reduces the number of computations required to calculate global feature dependencies. Extensive experiments have demonstrated the substantial superiority of this procedure over the most advanced techniques, suggesting that it has broad potential for clinical use.

    References

    [1]
    Hemant K. Aggarwal, Merry P. Mani, and Mathews Jacob. 2018. MoDL: Model-based deep learning architecture for inverse problems. IEEE Transactions on Medical Imaging 38, 2 (2018), 394–405.
    [2]
    Abdul Haseeb Ahmed, Ruixi Zhou, Yang Yang, Prashant Nagpal, Michael Salerno, and Mathews Jacob. 2020. Free-breathing and ungated dynamic mri using navigator-less spiral storm. IEEE Transactions on Medical Imaging 39, 12 (2020), 3933–3943.
    [3]
    Atif Alamri, Jongeun Cha, and Abdulmotaleb El Saddik. 2010. AR-REHAB: An augmented reality framework for poststroke-patient rehabilitation. IEEE Transactions on Instrumentation and Measurement 59, 10 (2010), 2554–2563.
    [4]
    Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. 2021. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 6836–6846.
    [5]
    Joao Carreira and Andrew Zisserman. 2017. Quo vadis, action recognition? A new model and the kinetics dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6299–6308.
    [6]
    Yuren Cong, Wentong Liao, Hanno Ackermann, Bodo Rosenhahn, and Michael Ying Yang. 2021. Spatial-temporal transformer for dynamic scene graph generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 16372–16382.
    [7]
    Jinglong Du, Zhongshi He, Lulu Wang, Ali Gholipour, Zexun Zhou, Dingding Chen, and Yuanyuan Jia. 2020. Super-resolution reconstruction of single anisotropic 3D MR images using residual convolutional neural network. Neurocomputing 392 (2020), 209–220.
    [8]
    Abdulmotaleb El Saddik. 2007. The potential of haptics technologies. IEEE Instrumentation and Measurement Magazine 10, 1 (2007), 10–17.
    [9]
    Abdulmotaleb El Saddik. 2018. Digital twins: The convergence of multimedia technologies. IEEE Multimedia 25, 2 (2018), 87–92.
    [10]
    Chun-Mei Feng, Yunlu Yan, Huazhu Fu, Li Chen, and Yong Xu. 2021. Task transformer network for joint MRI reconstruction and super-resolution. In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part VI 24. Springer, 307–317.
    [11]
    Rui Guo, Hossam El-Rewaidy, Salah Assana, Xiaoying Cai, Amine Amyar, Kelvin Chow, Xiaoming Bi, Tuyen Yankama, Julia Cirillo, Patrick Pierce, Beth Goddu, Long Ngo, and Reza Nezafat. 2022. Accelerated cardiac T1 mapping in four heartbeats with inline MyoMapNet: a deep learning-based T1 estimation approach. Journal of Cardiovascular Magnetic Resonance 24, 1 (2022), 1–15.
    [12]
    Xudong Guo, Xun Guo, and Yan Lu. 2021. Ssan: Separable self-attention network for video representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12618–12627.
    [13]
    Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. 2017. Learning spatio-temporal features with 3d residual networks for action recognition. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 3154–3160.
    [14]
    Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. 2019. Axial attention in multidimensional transformers. arXiv:1912.12180. Retrieved from https://arxiv.org/abs/1912.12180
    [15]
    M Shamim Hossain, Ghulam Muhammad, and Atif Alamri. 2019. Smart healthcare monitoring: A voice pathology detection paradigm for smart cities. Multimedia Systems 25 (2019), 565–575.
    [16]
    Qiaoying Huang, Dong Yang, Pengxiang Wu, Hui Qu, Jingru Yi, and Dimitris Metaxas. 2019. MRI reconstruction via cascaded channel-wise attention network. In Proceedings of the 2019 IEEE 16th International Symposium on Biomedical Imaging. IEEE, 1622–1626.
    [17]
    Hong Jung, Jong Chul Ye, and Eung Yeop Kim. 2007. Improved k–t BLAST and k–t SENSE using FOCUSS. Physics in Medicine and Biology 52, 11 (2007), 3201.
    [18]
    Guangyuan Li, Jun Lv, Yapeng Tian, Qi Dou, Chengyan Wang, Chenliang Xu, and Jing Qin. 2022. Transformer-empowered multi-scale contextual matching and aggregation for multi-contrast MRI super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 20636–20645.
    [19]
    Jingyun Liang, Jiezhang Cao, Yuchen Fan, Kai Zhang, Rakesh Ranjan, Yawei Li, Radu Timofte, and Luc Van Gool. 2022. Vrt: A video restoration transformer. arXiv:2201.12288. Retrieved from https://arxiv.org/abs/2201.12288
    [20]
    Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. 2021. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 1833–1844.
    [21]
    Jing Lin, Yuanhao Cai, Xiaowan Hu, Haoqian Wang, Youliang Yan, Xueyi Zou, Henghui Ding, Yulun Zhang, Radu Timofte, and Luc Van Gool. 2022. Flow-guided sparse transformer for video deblurring. In International Conference on Machine Learning. PMLR, 13334–13343.
    [22]
    Sajan Goud Lingala, Yue Hu, Edward DiBella, and Mathews Jacob. 2011. Accelerated dynamic MRI exploiting sparsity and low-rank structure: kt SLR. IEEE Transactions on Medical Imaging 30, 5 (2011), 1042–1054.
    [23]
    Guangming Wang, Jun Lyu, Fanwen Wang, Chengyan Wang, and Jing Qin. 2024. Multi-level temporal information sharing transformer-based feature reuse network for cardiac MRI reconstruction. In Statistical Atlases and Computational Models of the Heart. Regular and CMRxRecon Challenge Papers (STACOM’23), Oscar Camara, et al. (Eds.)., Lecture Notes in Computer Science, vol 14507. Springer, Cham.
    [24]
    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 10012–10022.
    [25]
    Jun Lv, Wenjian Huang, Jue Zhang, and Xiaoying Wang. 2018. Performance of U-net based pyramidal lucas-kanade registration on free-breathing multi-b-value diffusion MRI of the kidney. The British Journal of Radiology 91, 1086 (2018), 20170813.
    [26]
    Jun Lv, Guangyuan Li, Xiangrong Tong, Weibo Chen, Jiahao Huang, Chengyan Wang, and Guang Yang. 2021. Transfer learning enhanced generative adversarial networks for multi-channel MRI reconstruction. Computers in Biology and Medicine 134 (2021), 104504.
    [27]
    Jun Lv, Chengyan Wang, and Guang Yang. 2021. PIC-GAN: A parallel imaging coupled generative adversarial network for accelerated multi-channel MRI reconstruction. Diagnostics 11, 1 (2021), 61.
    [28]
    Jun Lv, Ming Yang, Jue Zhang, and Xiaoying Wang. 2018. Respiratory motion correction for free-breathing 3D abdominal MRI using CNN-based image registration: A feasibility study. The British Journal of Radiology 91, xxxx (2018), 20170788.
    [29]
    Jun Lyu, Guangyuan Li, Chengyan Wang, Chen Qin, Shuo Wang, Qi Dou, and Jing Qin. 2023. Region-focused multi-view transformer-based generative adversarial network for cardiac cine MRI reconstruction. Medical Image Analysis 85 (2023), 102760.
    [30]
    Jun Lyu, Bin Sui, Chengyan Wang, Yapeng Tian, Qi Dou, and Jing Qin. 2022. DuDoCAF: Dual-domain cross-attention fusion with recurrent transformer for fast multi-contrast MR imaging. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 474–484.
    [31]
    Balamurali Murugesan, S. Vijaya Raghavan, Kaushik Sarveswaran, Keerthi Ram, and Mohanasankar Sivaprakasam. 2019. Recon-glgan: A global-local context based generative adversarial network for mri reconstruction. In Machine Learning for Medical Image Reconstruction: 2nd International Workshop, MLMIR 2019, Held in Conjunction with MICCAI 2019, Shenzhen, China, October 17, 2019, Proceedings 2. Springer, 3–15.
    [32]
    Ricardo Otazo, Emmanuel Candes, and Daniel K. Sodickson. 2015. Low-rank plus sparse matrix decomposition for accelerated dynamic MRI with separation of background and dynamic components. Magnetic Resonance in Medicine 73, 3 (2015), 1125–1136.
    [33]
    AJ Piergiovanni, Weicheng Kuo, and Anelia Angelova. 2023. Rethinking video vits: Sparse video tubes for joint image and video learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2214–2224.
    [34]
    Chen Qin, Jo Schlemper, Jose Caballero, Anthony N. Price, Joseph V. Hajnal, and Daniel Rueckert. 2018. Convolutional recurrent neural networks for dynamic MR image reconstruction. IEEE Transactions on Medical Imaging 38, 1 (2018), 280–290.
    [35]
    Sriprabha Ramanarayanan, Balamurali Murugesan, Keerthi Ram, and Mohanasankar Sivaprakasam. 2020. DC-WCNN: A deep cascade of wavelet based convolutional neural networks for MR image reconstruction. In Proceedings of the 2020 IEEE 17th International Symposium on Biomedical Imaging. IEEE, 1069–1073.
    [36]
    Erik B. Schelbert and Daniel R. Messroghli. 2016. State of the art: Clinical applications of cardiac T1 mapping. Radiology 278, 3 (2016), 658–676.
    [37]
    Jo Schlemper, Jose Caballero, Joseph V. Hajnal, Anthony Price, and Daniel Rueckert. 2017. A deep cascade of convolutional neural networks for MR image reconstruction. In Information Processing in Medical Imaging: 25th International Conference, IPMI 2017, Boone, NC, USA, June 25-30, 2017, Proceedings 25. Springer, 647–658.
    [38]
    Karen Simonyan and Andrew Zisserman. 2014. Two-stream convolutional networks for action recognition in videos. Advances in Neural Information Processing Systems 27 (2014).
    [39]
    Andrew J. Taylor, Michael Salerno, Rohan Dharmakumar, and Michael Jerosch-Herold. 2016. T1 mapping: Basic techniques and clinical applications. JACC: Cardiovascular Imaging 9, 1 (2016), 67–81.
    [40]
    Alina L. Machidon and Veljko Pejovic. 2021. Deep learning techniques for compressive sensing-based reconstruction and inference–A ubiquitous systems perspective. arXiv preprint arXiv:2105.13191
    [41]
    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems 30 (2017).
    [42]
    Chengyan Wang, et al., 2023. CMRxRecon: An open cardiac MRI dataset for the competition of accelerated image reconstruction. arXiv preprint arXiv:2309.10836 (2023)
    [43]
    Xiaoqing Wang, Sebastian Rosenzweig, Volkert Roeloffs, Moritz Blumenthal, Nick Scholand, Zhengguo Tan, H. Christian M. Holme, Christina Unterberg-Buchwald, Rabea Hinkel, and Martin Uecker. 2023. Free-breathing myocardial T1 mapping using inversion-recovery radial FLASH and motion-resolved model-based reconstruction. Magnetic Resonance in Medicine 89, 4 (2023), 1368–1384.
    [44]
    Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. 2021. End-to-end video instance segmentation with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8741–8750.
    [45]
    Syed Umar Amin, Mansour Alsulaiman, Ghulam Muhammad, Mohamed Amine Mekhtiche, and M. Shamim Hossain. 2019. Deep Learning for EEG motor imagery classification based on multi-layer CNNs feature fusion, Future Generation Computer Systems, 101, (2019), 542–554.
    [46]
    Zhaohu Xing, Lequan Yu, Liang Wan, Tong Han, and Lei Zhu. 2022. NestedFormer: Nested modality-aware transformer for brain tumor segmentation. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 140–150.
    [47]
    Shen Yan, Xuehan Xiong, Anurag Arnab, Zhichao Lu, Mi Zhang, Chen Sun, and Cordelia Schmid. 2022. Multiview transformers for video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3333–3343.
    [48]
    Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. 2022. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10819–10829.

    Index Terms

    1. Iterative Temporal-spatial Transformer-based Cardiac T1 Mapping MRI Reconstruction

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Transactions on Multimedia Computing, Communications, and Applications
      ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 20, Issue 6
      June 2024
      715 pages
      ISSN:1551-6857
      EISSN:1551-6865
      DOI:10.1145/3613638
      • Editor:
      • Abdulmotaleb El Saddik
      Issue’s Table of Contents

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 08 March 2024
      Online AM: 29 January 2024
      Accepted: 24 January 2024
      Revised: 15 January 2024
      Received: 20 December 2023
      Published in TOMM Volume 20, Issue 6

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. Cardiac MRI reconstruction
      2. multi-level
      3. transformer
      4. temporal information
      5. T1 mapping

      Qualifiers

      • Research-article

      Funding Sources

      • Researchers Supporting Project number
      • King Saud University, Riyadh, Saudi Arabia
      • National Natural Science Foundation of China
      • Yantai Basic Research Key Project
      • Youth Innovation Science and Technology Support Program of Shandong Provincial

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • 0
        Total Citations
      • 182
        Total Downloads
      • Downloads (Last 12 months)182
      • Downloads (Last 6 weeks)11
      Reflects downloads up to 27 Jul 2024

      Other Metrics

      Citations

      View Options

      Get Access

      Login options

      Full Access

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Full Text

      View this article in Full Text.

      Full Text

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media