Abstract
For the first time, we propose using a multiple instance learning based convolution-free transformer model, called Multiple Instance Neuroimage Transformer (MINiT), for the classification of T1-weighted (T1w) MRIs. We first present several variants of transformer models adopted for neuroimages. These models extract non-overlapping 3D blocks from the input volume and perform multi-headed self-attention on a sequence of their linear projections. MINiT, on the other hand, treats each of the non-overlapping 3D blocks of the input MRI as its own instance, splitting it further into non-overlapping 3D patches, on which multi-headed self-attention is computed. As a proof-of-concept, we evaluate the efficacy of our model by training it to identify sex from T1w-MRIs of two public datasets: Adolescent Brain Cognitive Development (ABCD) and the National Consortium on Alcohol and Neurodevelopment in Adolescence (NCANDA). The learned attention maps highlight voxels contributing to identifying sex differences in brain morphometry. The code is available at https://github.com/singlaayush/MINIT.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Abnar, S., Zuidema, W.: Quantifying attention flow in transformers. arXiv:2005.00928 (2020)
Adeli, E., et al.: Deep learning identifies morphological determinants of sex differences in the pre-adolescent brain. Neuroimage 223, 117293 (2020)
Agarap, A.F.: Deep learning using rectified linear units (ReLU). arXiv:1803.08375 (2018)
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., Schmid, C.: ViViT: a video vision transformer. In: ICCV, pp. 6836–6846 (2021)
Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv:1607.06450 (2016)
Brown, S.A., et al.: The national consortium on alcohol and neurodevelopment in adolescence (NCANDA): a multisite study of adolescent development and substance use. JSAD 76(6), 895–908 (2015)
Carbonneau, M.A., Cheplygina, V., Granger, E., Gagnon, G.: Multiple instance learning: a survey of problem characteristics and applications. Pattern Recogn. 77, 329–353 (2018)
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 213–229. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_13
Casey, B., et al.: The adolescent brain cognitive development (ABCD) study: imaging acquisition across 21 sites. Dev. Cogn. Neurosci. 32, 43–54 (2018)
Chen, J., et al.: TransUNet: transformers make strong encoders for medical image segmentation. arXiv:2102.04306 (2021)
Chen, X., Hsieh, C.J., Gong, B.: When vision transformers outperform ResNets without pre-training or strong data augmentations. arXiv:2106.01548 (2021)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805 (2018)
Dosovitskiy, A., et al.: An image is worth \(16 \times 16\) words: transformers for image recognition at scale. arXiv:2010.11929 (2020)
Efraimidis, P.S., Spirakis, P.G.: Weighted random sampling with a reservoir. Inf. Process. Lett. 97(5), 181–185 (2006)
Goyal, P., et al.: Accurate, large minibatch SGD: training ImageNet in 1 hour. arXiv:1706.02677 (2017)
Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., Wang, Y.: Transformer in transformer. In: NeurIPS, vol. 34 (2021)
Hänggi, J., Buchmann, A., Mondadori, C.R., Henke, K., Jäncke, L., Hock, C.: Sexual dimorphism in the parietal substrate associated with visuospatial cognition independent of general intelligence. JoCN 22(1), 139–155 (2010)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)
Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: ICML, pp. 448–456. PMLR (2015)
Jun, E., Jeong, S., Heo, D.W., Suk, H.I.: Medical transformer: universal brain encoder for 3D MRI analysis. arXiv:2104.13633 (2021)
Kaczkurkin, A.N., Raznahan, A., Satterthwaite, T.D.: Sex differences in the developing brain: insights from multimodal neuroimaging. Neuropsychopharmacology 44(1), 71–85 (2019)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv:1412.6980 (2014)
Larrazabal, A.J., Nieto, N., Peterson, V., Milone, D.H., Ferrante, E.: Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis. Proc. Natl. Acad. Sci. 117(23), 12592–12594 (2020)
Liu, M., Zhang, J., Adeli, E., Shen, D.: Landmark-based deep multi-instance learning for brain disease diagnosis. Med. Image Anal. 43, 157–168 (2018)
Loshchilov, I., Hutter, F.: SGDR: stochastic gradient descent with warm restarts. arXiv:1608.03983 (2016)
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv:1711.05101 (2017)
Malkiel, I., Rosenman, G., Wolf, L., Hendler, T.: Pre-training and fine-tuning transformers for FMRI prediction tasks. arXiv:2112.05761 (2021)
Ouyang, J., et al.: Longitudinal pooling & consistency regularization to model disease progression from MRIs. IEEE J. Biomed. Health Inform. 25(6), 2082–2092 (2020)
Pohl, K.M., et al.: The ‘NCANDA_PUBLIC_6Y_STRUCTURAL_V01’ data release of the National Consortium on Alcohol and NeuroDevelopment in Adolescence (NCANDA). Sage Bionetworks Synapse (2022). https://doi.org/10.7303/syn32773308
Pramono, R.R.A., Chen, Y.T., Fang, W.H.: Hierarchical self-attention network for action localization in videos. In: ICCV, pp. 61–70 (2019)
Sacher, J., Neumann, J., Okon-Singer, H., Gotowiec, S., Villringer, A.: Sexual dimorphism in the human brain: evidence from neuroimaging. JMRI 31(3), 366–375 (2013)
Shazeer, N.: GLU variants improve transformer. arXiv:2002.05202 (2020)
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. JMLR 15(1), 1929–1958 (2014)
Steiner, A., Kolesnikov, A., Zhai, X., Wightman, R., Uszkoreit, J., Beyer, L.: How to train your ViT? data, augmentation, and regularization in vision transformers. arXiv:2106.10270 (2021)
Su, J., Lu, Y., Pan, S., Wen, B., Liu, Y.: RoFormer: enhanced transformer with rotary position embedding. arXiv:2104.09864 (2021)
Van Putten, M.J., Olbrich, S., Arns, M.: Predicting sex from brain rhythms with deep learning. Sci. Rep. 8(1), 1–7 (2018)
Vaswani, A., et al.: Attention is all you need. In: NeurIPS, vol. 30 (2017)
Wang, H., Zhu, Y., Adam, H., Yuille, A., Chen, L.C.: MaX-DeepLab: end-to-end panoptic segmentation with mask transformers. In: CVPR, pp. 5463–5474 (2021)
Xin, J., Zhang, Y., Tang, Y., Yang, Y.: Brain differences between men and women: evidence from deep learning. Front. Neurosci. 13, 185 (2019)
Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.: CutMix: regularization strategy to train strong classifiers with localizable features. In: ICCV, pp. 6023–6032 (2019)
Zhang, B., et al.: Co-training transformer with videos and images improves action recognition. arXiv:2112.07175 (2021)
Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: Mixup: beyond empirical risk minimization. arXiv:1710.09412 (2017)
Zhao, H., Jiang, L., Jia, J., Torr, P.H., Koltun, V.: Point transformer. In: ICCV, pp. 16259–16268 (2021)
Zhao, Q., Adeli, E., Pfefferbaum, A., Sullivan, E.V., Pohl, K.M.: Confounder-aware visualization of ConvNets. In: Suk, H.-I., Liu, M., Yan, P., Lian, C. (eds.) MLMI 2019. LNCS, vol. 11861, pp. 328–336. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32692-0_38
Acknowledgements
This work was partially supported by the NIH grants AA021697 and AA028840, and the Stanford Institute for Human-centered Artificial Intelligence (HAI) Google Cloud Credits (GCP) credits.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Singla, A., Zhao, Q., Do, D.K., Zhou, Y., Pohl, K.M., Adeli, E. (2022). Multiple Instance Neuroimage Transformer. In: Rekik, I., Adeli, E., Park, S.H., Cintas, C. (eds) Predictive Intelligence in Medicine. PRIME 2022. Lecture Notes in Computer Science, vol 13564. Springer, Cham. https://doi.org/10.1007/978-3-031-16919-9_4
Download citation
DOI: https://doi.org/10.1007/978-3-031-16919-9_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16918-2
Online ISBN: 978-3-031-16919-9
eBook Packages: Computer ScienceComputer Science (R0)
