Abstract
Multi-modal neuroimaging data, e.g., magnetic resonance imaging (MRI) and positron emission tomography (PET), has greatly advanced computer-aided diagnosis of Alzheimer’s disease (AD) and its prodromal stage, i.e., mild cognitive impairment (MCI). However, incomplete multi-modality data often limits the diagnostic performance of deep learning-based methods, as only partial data can be used for training neural networks, and meanwhile it is challenging to synthesize missing scans (e.g., PET) with meaningful patterns associated with AD. To this end, we propose a novel unified modality understanding network to directly extract discriminative features from incomplete multi-modal data for AD diagnosis. Specifically, the incomplete multi-modal neuroimages are first branched into the corresponding encoders to extract modality-specific features and a Transformer is then applied to adaptively fuse the incomplete multi-modal features for AD diagnosis. To alleviate the potential problem of domain shift due to incomplete multi-modal input, the cross-modality contrastive learning strategy is further leveraged to align the incomplete multi-modal features into a unified embedding space. On the other hand, the proposed network also employs inter-modality and intra-modality attention weights for achieving local- to-local and local-to-global attention consistency so as to better transfer the diagnostic knowledge from one modality to another. Meanwhile, we leverage multi-instance attention rectification to rectify the localization of AD-related atrophic area. Extensive experiments on ADNI datasets with 1,950 subjects demonstrate the superior performance of the proposed methods for AD diagnosis and MCI conversion prediction.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Association, A.: 2019 Alzheimer’s disease facts and figures. Alzheimer’s Dement. 15(3), 321–387 (2019)
Baron, J., et al.: In vivo mapping of gray matter loss with voxel-based morphometry in mild Alzheimer’s disease. Neuroimage 14(2), 298–309 (2001)
Brookmeyer, R., Johnson, E., Ziegler-Graham, K., Arrighi, H.M.: Forecasting the global burden of Alzheimer’s disease. Alzheimer’s Dement. 3(3), 186–191 (2007)
Dosovitskiy, A., et al.: An image is worth 16\(\,\times \,\)16 words: transformers for image recognition at scale. In: International Conference on Learning Representations (2021)
Gao, X., Shi, F., Shen, D., Liu, M.: Task-induced pyramid and attention GAN for multimodal brain image imputation and classification in Alzheimer’s disease. IEEE J. Biomed. Health Inform. 26(1), 36–43 (2022)
Han, K., He, M., Yang, F., Zhang, Y.: Multi-task multi-level feature adversarial network for joint Alzheimer’s disease diagnosis and atrophy localization using sMRI. Phys. Med. Biol. 67(8), 085002 (2022)
Holmes, C.J., Hoge, R., Collins, L., Evans, A.C.: Enhancement of t1 MR images using registration for signal averaging. NeuroImage 3(3, Supplement), S28 (1996)
Jenkinson, M., Bannister, P., Brady, M., Smith, S.: Improved optimization for the robust and accurate linear registration and motion correction of brain images. Neuroimage 17(2), 825–841 (2002)
Jenkinson, M., Smith, S.: A global optimisation method for robust affine registration of brain images. Med. Image Anal. 5(2), 143–156 (2001)
Klöppel, S., et al.: Automatic classification of MR scans in Alzheimer’s disease. Brain 131(3), 681–689 (2008)
Li, G., Nie, J., Wu, G., Wang, Y., Shen, D.: Consistent reconstruction of cortical surfaces from longitudinal brain MR images. Neuroimage 59(4), 3805–3820 (2012)
Li, W., et al.: UNIMO: towards unified-modal understanding and generation via cross-modal contrastive learning. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 2592–2607 (2021)
Liu, M., Zhang, J., Adeli, E., Shen, D.: Landmark-based deep multi-instance learning for brain disease diagnosis. Med. Image Anal. 43, 157–168 (2018)
Pan, Y., Liu, M., Lian, C., Xia, Y., Shen, D.: Spatially-constrained fisher representation for brain disease identification with incomplete multi-modal neuroimages. IEEE Trans. Med. Imaging 39(9), 2965–2975 (2020)
Pan, Y., Liu, M., Xia, Y., Shen, D.: Disease-image-specific learning for diagnosis-oriented neuroimage synthesis with incomplete multi-modality data. IEEE Trans. Pattern Anal. Mach. Intell. 44(10), 6839–6853 (2022)
Qiao, H., Chen, L., Ye, Z., Zhu, F.: Early Alzheimer’s disease diagnosis with the contrastive loss using paired structural MRIs. Comput. Methods Programs Biomed. 208, 106282 (2021)
Sled, J.G., Zijdenbos, A.P., Evans, A.C.: A nonparametric method for automatic correction of intensity nonuniformity in MRI data. IEEE Trans. Med. Imaging 17(1), 87–97 (1998)
Ségonne, F., et al.: A hybrid approach to the skull stripping problem in MRI. Neuroimage 22(3), 1060–1075 (2004)
Wang, L., et al.: Large deformation diffeomorphism and momentum based hippocampal shape discrimination in dementia of the Alzheimer type. IEEE Trans. Med. Imaging 26(4), 462–470 (2007)
Acknowledgements
Data used in the preparation of this article were obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database (adni.loni.usc.edu). As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not participate in the analysis or writing of this report. A complete listing of ADNI investigators can be found at:http://adni.loni.usc.edu/wp-content/uploads/how_to_ apply/ADNI_Acknowledgement_List.pdf. This work was supported in part by the National Natural Science Foundation of China under Grant 61771233, and Grant 61702182 to Feng Yang; in part by Guangdong Basic and Applied Basic Research Foundation under Grant No. 2023A1515011260, and Science and Technology Program of Guangzhou under Grant No. 202201011672 to Feng Yang.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Han, K., Zhao, F., Zhu, D., Liu, T., Yang, F., Li, G. (2024). Towards Unified Modality Understanding for Alzheimer’s Disease Diagnosis Using Incomplete Multi-modality Data. In: Cao, X., Xu, X., Rekik, I., Cui, Z., Ouyang, X. (eds) Machine Learning in Medical Imaging. MLMI 2023. Lecture Notes in Computer Science, vol 14349. Springer, Cham. https://doi.org/10.1007/978-3-031-45676-3_19
Download citation
DOI: https://doi.org/10.1007/978-3-031-45676-3_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-45675-6
Online ISBN: 978-3-031-45676-3
eBook Packages: Computer ScienceComputer Science (R0)
