Abstract
The accuracy of predictive models for solitary pulmonary nodule (SPN) diagnosis can be greatly increased by incorporating repeat imaging and medical context, such as electronic health records (EHRs). However, clinically routine modalities such as imaging and diagnostic codes can be asynchronous and irregularly sampled over different time scales which are obstacles to longitudinal multimodal learning. In this work, we propose a transformer-based multimodal strategy to integrate repeat imaging with longitudinal clinical signatures from routinely collected EHRs for SPN classification. We perform unsupervised disentanglement of latent clinical signatures and leverage time-distance scaled self-attention to jointly learn from clinical signatures expressions and chest computed tomography (CT) scans. Our classifier is pretrained on 2,668 scans from a public dataset and 1,149 subjects with longitudinal chest CTs, billing codes, medications, and laboratory tests from EHRs of our home institution. Evaluation on 227 subjects with challenging SPNs revealed a significant AUC improvement over a longitudinal multimodal baseline (0.824 vs 0.752 AUC), as well as improvements over a single cross-section multimodal scenario (0.809 AUC) and a longitudinal imaging-only scenario (0.741 AUC). This work demonstrates significant advantages with a novel approach for co-learning longitudinal imaging and non-imaging phenotypes with transformers. Code available at https://github.com/MASILab/lmsignatures.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ardila, D., et al.: End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography (2019). https://www.nature.com/articles/s41591-019-0447-x
Choi, E., Bahadori, M.T., Sun, J., Kulas, J., Schuetz, A., Stewart, W.: RETAIN: an interpretable predictive model for healthcare using reverse time attention mechanism. In: Advances in Neural Information Processing Systems. vol. 29 (2016)
Finch, A., et al.: Exploiting hierarchy in medical concept embedding. JAMIA Open 4(1), ooab022 (2021)
Fritsch, F.N., Butland, J.: A method for constructing local monotone piecewise cubic interpolants. SIAM J. Sci. Stat. Comput. 5(2), 300–304 (1984)
Gao, R., et al.: Time-distanced gates in long short-term memory networks. Med. Image Anal. 65(101785), 101785 (2020)
Gao, R., et al.: Deep multi-path network integrating incomplete biomarker and chest CT data for evaluating lung cancer risk. In: Medical Imaging 2021: Image Processing. vol. 11596, pp. 387–393. SPIE (2021)
Gaudet-Blavignac, C., Foufi, V., Bjelogrlic, M., Lovis, C.: Use of the Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT) for Processing Free Text in Health Care: Systematic Scoping Review. J. Med. Internet Res. 23(1), e24594 (2021)
Gómez-Sáez, N., et al.: Prevalence and variables associated with solitary pulmonary nodules in a routine clinic-based population: a cross-sectional study. Eur. Radiol. 24(9), 2174–2182 (2014)
Gould, M.K., et al.: Recent trends in the identification of incidental pulmonary nodules. Am. J. Respir. Crit. Care Med. 192(10), 1208–1214 (2015)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition (2015)
Huang, K., Altosaar, J., Ranganath, R.: Clinicalbert: Modeling clinical notes and predicting hospital readmission. arXiv:1904.05342 (2019)
Huang, P., et al.: Prediction of lung cancer risk at follow-up screening with low-dose CT: a training and validation study of a deep learning method. Lancet Dig. Health 1(7), e353–e362 (2019)
Hyvarinen, A.: Fast and robust fixed-point algorithms for independent component analysis. IEEE Trans. Neural Netw. 10(3), 626–634 (1999)
Labach, A., Pokhrel, A., Yi, S.E., Zuberi, S., Volkovs, M., Krishnan, R.G.: Effective self-supervised transformers for sparse time series data (2023). https://openreview.net/forum?id=HUCgU5EQluN
Lasko, T., et al.: EHR-driven machine-learning model to distinguish benign from malignant pulmonary nodules (2023)
Lasko, T.: Nonstationary gaussian process regression for evaluating repeated clinical laboratory tests. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 29 (2015)
Lasko, T.A.: Efficient inference of gaussian-process-modulated renewal processes with application to medical event data. In: Uncertainty in Artificial Intelligence: Proceedings of the Conference. Conference on Uncertainty in Artificial Intelligence. vol. 2014, p. 469. NIH Public Access (2014)
Lasko, T.A., Mesa, D.A.: Computational phenotype discovery via probabilistic independence. arXiv preprint arXiv:1907.11051 (2019)
Li, T.Z., et al.: Time-distance vision transformers in lung cancer diagnosis from longitudinal computed tomography. arXiv preprint arXiv:2209.01676 (2022)
Li, Y., et al.: BEHRT: transformer for electronic health records. Sci. Rep. 10(1), 1–12 (2020)
Liao, F., Liang, M., Li, Z., Hu, X., Song, S.: Evaluate the malignancy of pulmonary nodules using the 3D deep leaky noisy-or network. IEEE Trans. Neural Netw. Learn. Syst. 30(11), 3484–3495 (2019)
Massion, P.P., Walker, R.C.: Indeterminate pulmonary nodules: risk for having or for developing lung cancer? Cancer Prev. Res. (Phila) 7(12), 1173–1178 (2014)
McWilliams, A., et al.: Probability of cancer in pulmonary nodules detected on first screening CT. N. Engl. J. Med. 369(10), 910–919 (2013)
Mohsen, F., Ali, H., El Hajj, N., Shah, Z.: Artificial intelligence-based methods for fusion of electronic health records and imaging data. Sci. Rep. 12(1), 17981 (2022)
Rasmy, L., Xiang, Y., Xie, Z., Tao, C., Zhi, D.: Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. NPJ Dig. Med. 4(1), 86 (2021)
Rivera, M.P., Mehta, A.C., Wahidi, M.M.: Establishing the diagnosis of lung cancer: diagnosis and management of lung cancer, 3rd ed: American college of chest physicians evidence-based clinical practice guidelines. Chest 143(5 Suppl), e142S-e165S (2013)
Schütze, H., Manning, C.D., Raghavan, P.: Introduction to information retrieval. vol. 39. Cambridge University Press, Cambridge (2008)
Team, N.L.S.T.R.: Reduced lung-cancer mortality with low-dose computed tomographic screening. N. Engl. J. Med. 365(5), 395–409 (2011)
Tipirneni, S., Reddy, C.K.: Self-supervised transformer for sparse and irregularly sampled multivariate clinical time-series. ACM Trans. Knowl. Discov. Data (TKDD) 16(6), 1–17 (2022)
Vanguri, R.S., et al.: Multimodal integration of radiology, pathology and genomics for prediction of response to PD-(L) 1 blockade in patients with non-small cell lung cancer. Nat. Cancer 3(10), 1151–1164 (2022)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems. vol. 30 (2017)
Wu, C., Wu, F., Huang, Y.: Da-transformer: Distance-aware transformer. arXiv preprint arXiv:2010.06925 (2020)
Zhang, X., Zeman, M., Tsiligkaridis, T., Zitnik, M.: Graph-guided network for irregularly sampled multivariate time series. arXiv preprint arXiv:2110.05357 (2021)
Acknowledgements
This research was funded by the NIH through R01CA253923-02 and in part by NSF CAREER 1452485 and NSF 2040462. This research is also supported by ViSE through T32EB021937-07 and the Vanderbilt Institute for Clinical and Translational Research through UL1TR002243-06. We thank the National Cancer Institute for providing data collected through the NLST.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Li, T.Z. et al. (2023). Longitudinal Multimodal Transformer Integrating Imaging and Latent Clinical Signatures from Routine EHRs for Pulmonary Nodule Classification. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14221. Springer, Cham. https://doi.org/10.1007/978-3-031-43895-0_61
Download citation
DOI: https://doi.org/10.1007/978-3-031-43895-0_61
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43894-3
Online ISBN: 978-3-031-43895-0
eBook Packages: Computer ScienceComputer Science (R0)
