Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
SpringerLink
Log in

A Systematic Review of Multimodal Deep Learning Approaches for COVID-19 Diagnosis

  • Conference paper
  • First Online:
Image Analysis and Processing - ICIAP 2023 Workshops (ICIAP 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14366))

Included in the following conference series:

  • 344 Accesses

Abstract

During and after the years of the COVID-19 pandemic, researchers and domain experts put all their effort into the discovery of accurate and reliable techniques for the detection and diagnosis of this disease in potentially sick patients. In the meanwhile, Deep Learning (DL) techniques are continuously improving and expanding, becoming more and more efficient and compatible in several fields of study and with different kinds of data. This huge but heterogeneous set of data cannot be fully exploited if DL models are not designed to be compatible with different sources of data at the same time, therefore multimodal approaches were designed and adopted, resulting in better prediction results than the classic approaches. Given these premises, several multimodal solutions for COVID-19 diagnosis were built in these years, but it may result hard to have a complete overview of the current state-of-the-art. For this reason, this paper wants to be a useful review of multimodal approaches and related adopted datasets, and therefore a starting point to quickly check what to improve to bring more accurate solutions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Adamidi, E.S., Mitsis, K., Nikita, K.S.: Artificial intelligence in clinical care amidst COVID-19 pandemic: a systematic review. Comput. Struct. Biotechnol. J. 19, 2833–2850 (2021)

    Article  Google Scholar 

  2. Agricola, E., et al.: Heart and lung multimodality imaging in COVID-19. JACC Cardiovasc. Imaging 13(8), 1792–1808 (2020)

    Google Scholar 

  3. Almuayqil, S., Abd El-Ghany, S., Shehab, A.: Multimodality imaging of COVID-19 using fine-tuned deep learning models. Diagnostics 13(7), 1268 (2023)

    Article  Google Scholar 

  4. Almutairi, S.A.: A multimodal AI-based non-invasive COVID-19 grading framework powered by deep learning, manta ray, and fuzzy inference system from multimedia vital signs. Heliyon 9(6) (2023)

    Google Scholar 

  5. Asraf, A.: Covid19-pneumonia-normal-chest-xray-pa-dataset (kaggle) (2020)

    Google Scholar 

  6. Behrad, F., Saniee Abadeh, M.: An overview of deep learning methods for multimodal medical data mining. Expert Syst. Appl. 200, 117006 (2022). https://doi.org/10.1016/j.eswa.2022.117006, https://www.sciencedirect.com/science/article/pii/S0957417422004249

  7. Bertsimas, D., et al.: An aggregated dataset of clinical outcomes for COVID-19 patients (2020). https://www.covidanalytics.io/datasetdocumentation

  8. Born, J., et al.: On the role of artificial intelligence in medical imaging of COVID-19. Patterns 2(6) (2021)

    Google Scholar 

  9. Born, J., et al.: Pocovid-net: automatic detection of COVID-19 from a new lung ultrasound imaging dataset (pocus). arXiv preprint arXiv:2004.12084 (2020)

  10. Boulahia, S.Y., Amamra, A., Madi, M.R., Daikh, S.: Early, intermediate and late fusion strategies for robust deep learning-based multimodal action recognition. Mach. Vis. Appl. 32(6), 121 (2021)

    Article  Google Scholar 

  11. Cenggoro, T.W., Pardamean, B., et al.: A systematic literature review of machine learning application in COVID-19 medical image classification. Procedia Comput. Sci. 216, 749–756 (2023)

    Article  Google Scholar 

  12. Chetupalli, S.R., et al.: Multi-modal point-of-care diagnostics for COVID-19 based on acoustics and symptoms. IEEE J. Transl. Eng. Health Med. 11, 199–210 (2023)

    Article  Google Scholar 

  13. Cohen, J.P., Morrison, P., Dao, L.: COVID-19 image data collection. arXiv preprint arXiv:2003.11597 (2020)

  14. COVID, K.: Radiography database. Radiol. Soc. North Am. (2019). https://www.kaggle.com/tawsifurrahman/covid19-radiography-database. Accessed 1 Oct 2021

  15. Effati, M., Sun, Y.C., Naguib, H.E., Nejat, G.: Multimodal detection of COVID-19 symptoms using deep learning & probability-based weighting of modes. In: 2021 17th International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob), pp. 151–156. IEEE (2021)

    Google Scholar 

  16. Fahmy, G.A., Abd-Elrahman, E., Zorkany, M.: COVID-19 detection using multimodal and multi-model ensemble based deep learning technique. In: 2022 39th National Radio Science Conference (NRSC), vol. 1, pp. 241–253. IEEE (2022)

    Google Scholar 

  17. Filice, R.W., et al.: Crowdsourcing pneumothorax annotations using machine learning annotations on the NIH chest x-ray dataset. J. Digit. Imaging 33, 490–496 (2020)

    Article  Google Scholar 

  18. Guarrasi, V., et al.: Multimodal explainability via latent shift applied to COVID-19 stratification. arXiv preprint arXiv:2212.14084 (2022)

  19. Hammad, M., et al.: Efficient multimodal deep-learning-based COVID-19 diagnostic system for noisy and corrupted images. J. King Saud Univ.-Sci. 34(3), 101898 (2022)

    Article  Google Scholar 

  20. Hilmizen, N., Bustamam, A., Sarwinda, D.: The multimodal deep learning for diagnosing COVID-19 pneumonia from chest CT-scan and x-ray images. In: 2020 3rd International Seminar on Research of Information Technology and Intelligent Systems (ISRITI), pp. 26–31. IEEE (2020)

    Google Scholar 

  21. Horry, M.J., et al.: COVID-19 detection through transfer learning using multimodal imaging data. IEEE Access 8, 149808–149824 (2020)

    Article  Google Scholar 

  22. Jayachitra, V., Nivetha, S., Nivetha, R., Harini, R.: A cognitive IoT-based framework for effective diagnosis of COVID-19 using multimodal data. Biomed. Sig. Process. Control 70, 102960 (2021)

    Article  Google Scholar 

  23. Khanzada, A., Wilson, T.: Virufy COVID-19 open cough dataset, github (2020) (2021)

    Google Scholar 

  24. Kumar, S., et al.: A novel multimodal fusion framework for early diagnosis and accurate classification of COVID-19 patients using x-ray images and speech signal processing techniques. Comput. Methods Programs Biomed. 226, 107109 (2022)

    Article  Google Scholar 

  25. Larici, A.R., et al.: Multimodality imaging of COVID-19 pneumonia: from diagnosis to follow-up. A comprehensive review. Eur. J. Radiol. 131, 109217 (2020)

    Google Scholar 

  26. Li, Y., et al.: Automated multi-view multi-modal assessment of COVID-19 patients using reciprocal attention and biomedical transform. Front. Publ. Health 10, 886958 (2022)

    Article  Google Scholar 

  27. Lu, Y., Niu, K., Peng, X., Zeng, J., Pei, S.: Multi-modal intermediate fusion model for diagnosis prediction. In: 2022 the 6th International Conference on Innovation in Artificial Intelligence (ICIAI). ICIAI 2022, pp. 38–43. Association for Computing Machinery, New York, NY, USA (2022). https://doi.org/10.1145/3529466.3529496

  28. Maftouni, M., Law, A.C.C., Shen, B., Grado, Z.J.K., Zhou, Y., Yazdi, N.A.: A robust ensemble-deep learning model for COVID-19 diagnosis based on an integrated CT scan images database. In: IIE Annual Conference. Proceedings, pp. 632–637. Institute of Industrial and Systems Engineers (IISE) (2021)

    Google Scholar 

  29. Mahalle, P.N., Sable, N.P., Mahalle, N.P., Shinde, G.R.: Data analytics: COVID-19 prediction using multimodal data. In: Intelligent Systems and Methods to Combat Covid-19, pp. 1–10 (2020)

    Google Scholar 

  30. Mathieu, E., et al.: Coronavirus pandemic (COVID-19). Our World in Data (2020). https://ourworldindata.org/coronavirus

  31. Mayya, V., Karthik, K., Sowmya, K.S., Karadka, K., Jeganathan, J.: COVIDdx: AI-based clinical decision support system for learning COVID-19 disease representations from multimodal patient data. In: HEALTHINF, pp. 659–666 (2021)

    Google Scholar 

  32. Mukhi, S.E., Varshini, R.T., Sherley, S.E.F.: Diagnosis of COVID-19 from multimodal imaging data using optimized deep learning techniques. SN Comput. Sci. 4(3), 212 (2023)

    Article  Google Scholar 

  33. Nasir, N., et al.: Multi-modal image classification of COVID-19 cases using computed tomography and x-rays scans. Intell. Syst. Appl. 17, 200160 (2023)

    Google Scholar 

  34. Nguyen-Trong, K., Nguyen-Hoang, K.: Multi-modal approach for COVID-19 detection using coughs and self-reported symptoms. J. Intell. Fuzzy Syst. (Preprint), 1–13 (2023)

    Google Scholar 

  35. Niu, K., Zhang, K., Peng, X., Pan, Y., Xiao, N.: Deep multi-modal intermediate fusion of clinical record and time series data in mortality prediction. Front. Mol. Biosci. 10, 1136071 (2023)

    Article  Google Scholar 

  36. Orlandic, L., Teijeiro, T., Atienza, D.: The coughvid crowdsourcing dataset, a corpus for the study of large-scale cough analysis algorithms. Sci. Data 8(1), 156 (2021)

    Article  Google Scholar 

  37. Padmapriya, T., Kalaiselvi, T., Priyadharshini, V.: Multimodal COVID network: multimodal bespoke convolutional neural network architectures for COVID-19 detection from chest x-ray’s and computerized tomography scans. Int. J. Imaging Syst. Technol. 32(3), 704–716 (2022)

    Article  Google Scholar 

  38. Page, M.J., et al.: The Prisma 2020 statement: an updated guideline for reporting systematic reviews. Int. J. Surg. 88, 105906 (2021)

    Article  Google Scholar 

  39. Praveen: Coronahack -chest x-ray-dataset (kaggle) (2020)

    Google Scholar 

  40. Qayyum, A., Ahmad, K., Ahsan, M.A., Al-Fuqaha, A., Qadir, J.: Collaborative federated learning for healthcare: multi-modal COVID-19 diagnosis at the edge. IEEE Open J. Comput. Soc. 3, 172–184 (2022)

    Article  Google Scholar 

  41. Rahman, M.A., Hossain, M.S., Alrajeh, N.A., Gupta, B.: A multimodal, multimedia point-of-care deep learning framework for COVID-19 diagnosis. ACM Trans. Multimidia Comput. Commun. Appl. 17(1s), 1–24 (2021)

    Article  Google Scholar 

  42. Rashid, H.A., Sajadi, M.M., Mohsenin, T.: Coughnet-v2: a scalable multimodal DNN framework for point-of-care edge devices to detect symptomatic covid-19 cough. In: 2022 IEEE Healthcare Innovations and Point of Care Technologies (HI-POCT), pp. 37–40. IEEE (2022)

    Google Scholar 

  43. Hu, R., Ruan, G., Xiang, S., Huang, M., Liang, Q., Li, J.: Automated diagnosis of COVID-19 using deep learning and data augmentation on chest CT. medRxiv (2020)

    Google Scholar 

  44. Sait, U., et al.: A deep-learning based multimodal system for COVID-19 diagnosis using breathing sounds and chest x-ray images. Appl. Soft Comput. 109, 107522 (2021)

    Google Scholar 

  45. Sekaran, K., Gnanasambandan, R., Thirunavukarasu, R., Iyyadurai, R., Karthick, G., Doss, C.G.P.: A systematic review of artificial intelligence-based COVID-19 modeling on multimodal genetic information. Progr. Biophys. Mol. Biol. (2023)

    Google Scholar 

  46. Sharma, N., et al.: Coswara - a database of breathing, cough, and voice sounds for COVID-19 diagnosis. arXiv preprint arXiv:2005.10548 (2020)

  47. Sikkandar, M.Y., et al.: Leveraging multimodal ensemble fusion-based deep learning for COVID-19 on chest radiographs. Comput. Syst. Sci. Eng. 47(1) (2023)

    Google Scholar 

  48. Soares, E., Angelov, P., Biaso, S., Froes, M.H., Abe, D.K.: Sars-cov-2 CT-scan dataset: a large dataset of real patients CT scans for sars-cov-2 identification. medRxiv, pp. 2020–04 (2020)

    Google Scholar 

  49. Soda, P., et al.: AiforCOVID: predicting the clinical outcomes in patients with COVID-19 applying AI to chest-x-rays. An Italian multicentre study. Med. Image Anal. 74, 102216 (2021)

    Google Scholar 

  50. Stahlschmidt, S.R., Ulfenborg, B., Synnergren, J.: Multimodal deep learning for biomedical data fusion: a review. Brief. Bioinform. 23(2), bbab569 (2022)

    Google Scholar 

  51. Tabik, S., et al.: CovidGR dataset and COVID-SDNET methodology for predicting COVID-19 based on chest x-ray images. IEEE J. Biomed. Health Inform. 24(12), 3595–3605 (2020). https://doi.org/10.1109/JBHI.2020.3037127

    Article  Google Scholar 

  52. Tang, S., Hu, X., Atlas, L., Khanzada, A., Pilanci, M.: Hierarchical multi-modal transformer for automatic detection of COVID-19. In: Proceedings of the 2022 5th International Conference on Signal Processing and Machine Learning, pp. 197–202 (2022)

    Google Scholar 

  53. Varadarajan, V., Shabani, M., Ambale Venkatesh, B., Lima, J.A.: Role of imaging in diagnosis and management of COVID-19: a multiorgan multimodality imaging review. Front. Med. 8, 765975 (2021)

    Article  Google Scholar 

  54. Xia, T., et al.: COVID-19 sounds: a large-scale audio dataset for digital respiratory screening. In: Thirty-Fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (round 2) (2021)

    Google Scholar 

  55. Zhao, J., Zhang, Y., He, X., Xie, P.: COVID-CT-dataset: a CT scan dataset about COVID-19. arXiv preprint arXiv:2003.13865 490(10.48550) (2020)

Download references

Acknowledgements

This work is part of the POR FESR CAMPANIA 2014-2020 Synergy for COVID project (CUP H69I22000710002).

Author information

Authors and Affiliations

  1. Department of Electrical Engineering and Information Technologies, University of Naples Federico II, Via Claudio 21, 80125, Naples, Italy

    Salvatore Capuozzo & Carlo Sansone

Authors
  1. Salvatore Capuozzo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Carlo Sansone
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Salvatore Capuozzo .

Editor information

Editors and Affiliations

  1. University of Udine, Udine, Italy

    Gian Luca Foresti

  2. University of Udine, Udine, Italy

    Andrea Fusiello

  3. University of York, York, UK

    Edwin Hancock

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Capuozzo, S., Sansone, C. (2024). A Systematic Review of Multimodal Deep Learning Approaches for COVID-19 Diagnosis. In: Foresti, G.L., Fusiello, A., Hancock, E. (eds) Image Analysis and Processing - ICIAP 2023 Workshops. ICIAP 2023. Lecture Notes in Computer Science, vol 14366. Springer, Cham. https://doi.org/10.1007/978-3-031-51026-7_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-51026-7_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-51025-0

  • Online ISBN: 978-3-031-51026-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation