Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

Visualizing Salient Network Activations in Convolutional Neural Networks for Medical Image Modality Classification

  • Conference paper
  • First Online:
Recent Trends in Image Processing and Pattern Recognition (RTIP2R 2018)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1036))

Abstract

Convolutional neural network (CNN) has become the architecture of choice for visual recognition tasks. However, these models are perceived as black boxes since there is a lack of understanding of their learned behavior from the underlying task of interest. This lack of transparency is a drawback since poorly understood model behavior could adversely impact subsequent decision-making. Researchers use novel machine learning (ML) tools to classify the medical imaging modalities. However, it is poorly understood how these algorithms discriminate the modalities and if there are implicit opportunities for improving visual information access applications in computational biomedicine. In this study, we visualize the learned weights and salient network activations in a CNN based Deep Learning (DL) model to determine the image characteristics that lend themselves for improved classification with a goal of developing informed clinical question-answering systems. To support our analysis we cross-validate model performance to reduce bias and generalization errors and perform statistical analyses to assess performance differences.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Ben Abacha, A., Gayen, S., Lau, J.J., Rajaraman, S., Demner-Fushman, D.: NLM at ImageCLEF 2018 visual question answering in the medical domain. In: CEUR Workshop Proceedings, p. 2125 (2018)

    Google Scholar 

  2. Demner-Fushman, D., Antani, S., Thoma, G.R., Simpson, M.: Design and development of a multimodal biomedical information retrieval system. J. Comput. Sci. Eng. 6, 168–177 (2012)

    Article  Google Scholar 

  3. Rajaraman, S., Candemir, S., Kim, I., Thoma, G.R., Antani, S.: Visualization and interpretation of convolutional neural network predictions in detecting pneumonia in pediatric chest radiographs. MDPI Appl. Sci. 8(10), 1715 (2018)

    Article  Google Scholar 

  4. Rajaraman, S., et al.: Understanding the learned behavior of customized convolutional neural networks toward malaria parasite detection in thin blood smear images. J. Med. Imag. 5(3), 034501 (2018)

    Article  Google Scholar 

  5. Rajaraman, S., et al.: A novel stacked generalization of models for improved TB detection in chest radiographs. In: Proceedings of the International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 718–721 (2018)

    Google Scholar 

  6. Thamizhvani, T.R., Lakshmanan, S., Rajaraman, S.: Mobile application-based computer-aided diagnosis of skin tumours from dermal images. Imaging Sci J. 66(6), 382–391 (2018)

    Article  Google Scholar 

  7. Khan, S., Yong, S.P.: A comparison of deep learning and hand crafted features in medical image modality classification. In: Proceedings of the International Conference on Computer and Information Sciences, pp. 633–638 (2016)

    Google Scholar 

  8. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436–444 (2015)

    Article  Google Scholar 

  9. Rajaraman, S., et al.: Pre-trained convolutional neural networks as feature extractors toward improved malaria parasite detection in thin blood smear images. PeerJ 6, e4568 (2018)

    Article  Google Scholar 

  10. Rajaraman, S., et al.: Comparing deep learning models for population screening using chest radiography. In: Proceedings of the SPIE Medical Imaging: Computer-aided Diagnosis, p. 105751E (2018)

    Google Scholar 

  11. Rajaraman, S., Antani, S., Xue, Z., Candemir, S., Jaeger, S., Thoma, G.R.: Visualizing abnormalities in chest radiographs through salient network activations in deep learning. In: Proceedings of the IEEE Life Sciences Conference, pp. 71–74 (2017)

    Google Scholar 

  12. Rajaraman, S., Antani, S., Jaeger, S.: Visualizing deep learning activations for improved malaria cell classification. Proc. Mach. Learn. Res. 69, 40–47 (2017)

    Google Scholar 

  13. Xue, Z., Rajaraman, S., Long, L.R., Antani, S., Thoma, G.R.: Gender detection from spine x-ray images using deep learning. In: Proceedings of the IEEE International Symposium on Computer-based Medical Systems, pp. 54–58 (2018)

    Google Scholar 

  14. Simard, P., Steinkraus, D., Platt, J.C.: Best practices for convolutional neural networks applied to visual document analysis. In: Proceedings of the 7th International Conference on Document Analysis and Recognition, pp. 958–963 (2003)

    Google Scholar 

  15. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Proceedings of the Advances in Neural Information Processing Systems, pp. 1–9 (2012)

    Google Scholar 

  16. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Li, F.F.: ImageNet: a large-scale hierarchical image database. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009)

    Google Scholar 

  17. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2015)

  18. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826 (2016)

    Google Scholar 

  19. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  20. Chollet, F.: Xception: Deep Learning with Separable Convolutions. arXiv preprint arXiv:1610.02357 (2016)

  21. Huang, G., Liu, Z., Weinberger, K.Q., van der Maaten, L.: Densely Connected Convolutional Networks. arXiv preprint arXiv:1608.06993 (2017)

  22. Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015)

    Article  Google Scholar 

  23. Margeta, J., Criminisi, A., Lozoya, R.C., Lee, D.C., Ayache, N.: Fine-tuned convolutional neural nets for cardiac MRI acquisition plane recognition. Comput. Methods Biomech. Biomed. Eng. Imaging Vis. 5, 339–349 (2017)

    Article  Google Scholar 

  24. Lynch, S., Ng, A.: Why AI is the new electricity. https://news.stanford.edu/thedish/2017/03/14/andrew-ng-why-ai-is-the-new-electricity/

  25. Razavian, A.S., Azizpour, H., Sullivan, J., Carlsson, S.: CNN features off-the-shelf: an astounding baseline for recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 512–519 (2014)

    Google Scholar 

  26. De Herrera, A., Schaer, R., Bromuri, S., Müller, H.: Overview of the ImageCLEF 2016 medical task. In: CEUR Workshop Proceedings, p. 1609 (2016)

    Google Scholar 

  27. Apostolova, E., You, D., Xue, Z., Antani, S., Demner-Fushman, D., Thoma, G.R.: Image retrieval from scientific publications: Text and image content processing to separate multipanel figures. J. Am. Soc. Inf. Sci. Tec. 64, 893–908 (2013)

    Article  Google Scholar 

  28. Santosh, K.C., Aafaque, A., Antani, S., Thoma, G.R.: Line segment-based stitched multipanel figure separation for effective biomedical CBIR. Int. J. Pattern Recogn. Artif. Intell. 31(6), 1757003 (2017)

    Article  Google Scholar 

  29. Santosh, K.C., Xue, Z., Antani, S., Thoma, G.R.: NLM at ImageCLEF 2015: biomedical multipanel figure separation. In: CEUR Workshop Proceedings, p. 1391 (2015)

    Google Scholar 

  30. Santosh, K.C., Antani, S., Thoma, G.R.: Stitched multipanel biomedical figure separation. In: IEEE International Symposium on Computer-based Medical Systems, pp. 54–59 (2009)

    Google Scholar 

  31. De Herrera, A., Markonis, D., Müller, H.: Bag–of–colors for biomedical document image classification. In: Greenspan, H., Müller, H., Syeda-Mahmood, T. (eds.) MCBR-CDS 2012. LNCS, vol. 7723, pp. 110–121. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-36678-9_11

    Chapter  Google Scholar 

  32. Pelka, O., Friedrich, C.M.: FHDO biomedical computer science group at medical classification task of ImageCLEF 2015. In: CEUR Workshop Proceedings, p. 1391 (2015)

    Google Scholar 

  33. Cirujeda, P., Binefa, X.: Medical image classification via 2D color feature based covariance descriptors. In: CEUR Workshop Proceedings, p. 1391 (2015)

    Google Scholar 

  34. Li, P., et al.: UDEL CIS at ImageCLEF medical task 2016. In: CEUR Workshop Proceedings, p. 1609 (2016)

    Google Scholar 

  35. De Herrera, A., Kalpathy-Cramer, J., Fushman, D.D., Antani, S., Müller, H.: Overview of the imageCLEF 2013 medical tasks. In: CEUR Workshop Proceedings, p. 1179 (2013)

    Google Scholar 

  36. Yu, Y., et al.: Modality classification for medical images using multiple deep convolutional neural networks. J. Comput. Inf. Syst. 11(15), 5403–5413 (2015)

    Google Scholar 

  37. Kumar, A., Kim, J., Lyndon, D., Fulham, M., Feng, D.: An ensemble of fine-tuned convolutional neural networks for medical image classification. IEEE J. Biomed. Heal. Inf. 21, 31–40 (2017)

    Article  Google Scholar 

  38. Yu, Y., Lin, H., Meng, J., Wei, X., Guo, H., Zhao, Z.: Deep transfer learning for modality classification of medical images. MDPI Inf. 8(3), 91 (2017)

    Google Scholar 

  39. Koitka, S., Friedrich, C.M.: Traditional feature engineering and deep learning approaches at medical classification task of ImageCLEF 2016. In: CEUR Workshop Proceedings, p. 1609 (2016)

    Google Scholar 

  40. Zhang, J., Xia, Y., Wu, Q., Xie, Y.: Classification of Medical Images and Illustrations in the Biomedical Literature Using Synergic Deep Learning. arXiv preprint arXiv:1706.09092 (2017)

  41. Vallières, M., Freeman, C.R., Skamene, S.R., El Naqa, I.: A radiomics model from joint FDG-PET and MRI texture features for the prediction of lung metastases in soft-tissue sarcomas of the extremities. Phys. Med. Biol. 60, 5471–5496 (2015)

    Article  Google Scholar 

  42. Bloch, B., Jain, A., Jaffe, C.: Data From BREAST-DIAGNOSIS. https://wiki.cancerimagingarchive.net/display/Public/BREAST-DIAGNOSIS#9e4592af79b249bfaff992eceebbf842

  43. Vallières, M., et al.: Radiomics strategies for risk assessment of tumour failure in head-and-neck cancer. Sci. Rep. 7(1), 10117 (2017)

    Article  Google Scholar 

  44. Gevaert, O., et al.: Non-small cell lung cancer: identifying prognostic imaging biomarkers by leveraging public gene expression microarray data-methods and preliminary results. Radiology 264, 387–396 (2012)

    Article  Google Scholar 

  45. Kurdziel, K.A., et al.: The kinetics and reproducibility of 18F-sodium fluoride for oncology using current pet camera technology. J. Nucl. Med. 53, 1175–1184 (2012)

    Article  Google Scholar 

  46. Clark, K., et al.: The cancer imaging archive (TCIA): maintaining and operating a public information repository. J. Digit. Imaging 26, 1045–1057 (2013)

    Article  Google Scholar 

  47. Decencière, E., et al.: Feedback on a publicly distributed image database: the messidor database. Image Anal. Stereol. 33, 231–234 (2014)

    Article  Google Scholar 

  48. Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R.: ChestX-ray8: hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–19 (2017)

    Google Scholar 

  49. Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. J. Mach. Learn. Res. 13, 281–305 (2012)

    MathSciNet  MATH  Google Scholar 

  50. Matthews, B.W.: Comparison of the predicted and observed secondary structure of T4 phage lysozyme. BBA - Protein Struct. 405, 442–451 (1975)

    Article  Google Scholar 

  51. Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) Computer Vision – ECCV 2014, LNCS, vol. 8689, pp. 818–833. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10590-1_53

    Chapter  Google Scholar 

  52. Groeneveld, R.A., Meeden, G.: Measuring skewness and kurtosis. Statistician 33, 391–399 (1984)

    Article  Google Scholar 

  53. Rossi, J.S.: One-way ANOVA from summary statistics. Educ. Psychol. Meas. 47, 37–38 (1987)

    Article  Google Scholar 

  54. Daya, S.: One-way analysis of variance. Evid. Based Obstet. Gynecol. 5, 153–155 (2003)

    Article  Google Scholar 

  55. Shapiro, S.S., Wilk, M.B.: An analysis of variance test for normality (complete samples). Biometrika 52, 591 (1965)

    Article  MathSciNet  Google Scholar 

  56. Gastwirth, J.L., Gel, Y.R., Miao, W.: The Impact of levene’s test of equality of variances on statistical theory and practice. Stat. Sci. 24, 343–360 (2009)

    Article  MathSciNet  Google Scholar 

  57. Kucuk, U., Eyuboglu, M., Kucuk, H.O., Degirmencioglu, G.: Importance of using proper post hoc test with ANOVA. Int. J. Cardiol. 209, 346 (2016)

    Article  Google Scholar 

  58. Bressler, S.L.: Large-scale cortical networks and cognition. Brain Res. Rev. 20(3), 288–304 (1995)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Lister Hill National Center for Biomedical Communications, National Library of Medicine, Bethesda, MD, 20894, USA

    Sivaramakrishnan Rajaraman & Sameer Antani

Authors
  1. Sivaramakrishnan Rajaraman
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sameer Antani
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sivaramakrishnan Rajaraman .

Editor information

Editors and Affiliations

  1. Department of Computer Science, University of South Dakota, Vermillion, SD, USA

    K. C. Santosh

  2. Solapur University, Solapur, India

    Ravindra S. Hegadi

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Rajaraman, S., Antani, S. (2019). Visualizing Salient Network Activations in Convolutional Neural Networks for Medical Image Modality Classification. In: Santosh, K., Hegadi, R. (eds) Recent Trends in Image Processing and Pattern Recognition. RTIP2R 2018. Communications in Computer and Information Science, vol 1036. Springer, Singapore. https://doi.org/10.1007/978-981-13-9184-2_4

Download citation

  • DOI: https://doi.org/10.1007/978-981-13-9184-2_4

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-13-9183-5

  • Online ISBN: 978-981-13-9184-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics