Abstract
Convolutional neural network based systems have largely failed to be adopted in many high-risk application areas, including healthcare, military, security, transportation, finance, and legal, due to their highly uninterpretable “black-box” nature. Towards solving this deficiency, we teach a novel multi-task capsule network to improve the explainability of predictions by embodying the same high-level language used by human-experts. Our explainable capsule network, X-Caps, encodes high-level visual object attributes within the vectors of its capsules, then forms predictions based solely on these human-interpretable features. To encode attributes, X-Caps utilizes a new routing sigmoid function to independently route information from child capsules to parents. Further, to provide radiologists with an estimate of model confidence, we train our network on a distribution of expert labels, modeling inter-observer agreement and punishing over/under confidence during training, supervised by human-experts’ agreement. X-Caps simultaneously learns attribute and malignancy scores from a multi-center dataset of over 1000 CT scans of lung cancer screening patients. We demonstrate a simple 2D capsule network can outperform a state-of-the-art deep dense dual-path 3D CNN at capturing visually-interpretable high-level attributes and malignancy prediction, while providing malignancy prediction scores approaching that of non-explainable 3D CNNs. To the best of our knowledge, this is the first study to investigate capsule networks for making predictions based on radiologist-level interpretable attributes and its applications to medical image diagnosis. Code is publicly available at https://github.com/lalonderodney/X-Caps.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Afshar, P., Mohammadi, A., Plataniotis, K.N.: Brain tumor type classification via capsule networks. In: 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 3129–3133. IEEE (2018)
Armato III, S., et al.: The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): a completed reference database of lung nodules on CT scans. Med. Phys. 38(2), 915–931 (2011)
Bloomberg, J.: Don’t Trust Artificial Intelligence? Time To Open The AI ‘Black Box’, (11162018), Forbes Magazine. http://www.forbes.com/sites/jasonbloomberg/2018/09/16/dont-trust-artificial-intelligence-time-to-open-the-ai-black-box/#6ceaf3793b4a
Bologna, G.: A model for single and multiple knowledge based networks. Artif. Intell. Med. 28(2), 141–163 (2003)
Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., Abbeel, P.: Infogan: interpretable representation learning by information maximizing generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2172–2180 (2016)
Frankle, J., Carbin, M.: The lottery ticket hypothesis: finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635 (2018)
Gilpin, L.H., Bau, D., Yuan, B.Z., Bajwa, A., Specter, M., Kagal, L.: Explaining explanations: an overview of interpretability of machine learning. In: 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), pp. 80–89. IEEE (2018)
Hancock, M., Magnan, J.: Lung nodule malignancy classification using only radiologist-quantified image features as inputs to statistical learning algorithms. J. Med. Imaging 3(4), 044504 (2016)
Hussein, S., Cao, K., Song, Q., Bagci, U.: Risk stratification of lung nodules using 3D CNN-based multi-task learning. In: Niethammer, M., et al. (eds.) IPMI 2017. LNCS, vol. 10265, pp. 249–260. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-59050-9_20
Hussein, S., Gillies, R., Cao, K., Song, Q., Bagci, U.: Tumornet: lung nodule characterization using multi-view convolutional neural network with gaussian process. In: 14th International Symposium on Biomedical Imaging (ISBI), pp. 1007–1010. IEEE (2017)
Iesmantas, T., Alzbutas, R.: Convolutional capsule network for classification of breast cancer histology images. In: Campilho, A., Karray, F., ter Haar Romeny, B. (eds.) ICIAR 2018. LNCS, vol. 10882, pp. 853–860. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-93000-8_97
Jiménez-Sánchez, A., Albarqouni, S., Mateus, D.: Capsule networks against medical imaging data challenges. In: Stoyanov, D., et al. (eds.) LABELS/CVII/STENT -2018. LNCS, vol. 11043, pp. 150–160. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01364-6_17
Kandel, P., LaLonde, R., Ciofoaia, V., Wallace, M.B., Bagci, U.: Su1741 colorectal polyp diagnosis with contemporary artificial intelligence. Gastrointest. Endosc. 89(6), AB403 (2019)
Kuhn, M., Johnson, K.: Applied Predictive Modeling, vol. 26. Springer, New York (2013). https://doi.org/10.1007/978-1-4614-6849-3
LaLonde, R., Bagci, U.: Capsules for object segmentation. arXiv preprint arXiv:1804.04241 (2018)
LaLonde, R., Kandel, P., Spampinato, C., Wallace, M.B., Bagci, U.: Diagnosing colorectal polyps in the wild with capsule networks. In: 17th International Symposium on Biomedical Imaging (ISBI). IEEE (2020)
LaLonde, R., Xu, Z., Jain, S., Bagci, U.: Capsules for biomedical image segmentation. arXiv preprint arXiv:2004.04736 (2020)
Lehnis, M.: Can We Trust AI If We Don’t Know How It Works? (15062018), BBC News. http://www.bbc.com/news/business-44466213
Mobiny, A., Lu, H., Nguyen, H.V., Roysam, B., Varadarajan, N.: Automated classification of apoptosis in phase contrast microscopy using capsule network. IEEE Trans. Med. Imaging 39(1), 1–10 (2019)
Mobiny, A., Van Nguyen, H.: Fast CapsNet for lung cancer screening. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11071, pp. 741–749. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00934-2_82
Nibali, A., He, Z., Wollersheim, D.: Pulmonary nodule classification with deep residual networks. Int. J. Comput. Assist. Radiol. Surgery 1–10 (2017). https://doi.org/10.1007/s11548-017-1605-6
Pal, A., Chaturvedi, A., Garain, U., Chandra, A., Chatterjee, R., Senapati, S.: CapsDeMM: capsule network for detection of munro’s microabscess in skin biopsy images. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11071, pp. 389–397. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00934-2_44
Polonski, V.: People Don’t Trust AI-Here’s How We Can Change That, (10012018), Scientific American. http://www.scientificamerican.com/article/people-dont-trust-ai-heres-how-we-can-change-that/
Rudin, C.: Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. 1(5), 206–215 (2019)
Sabour, S., Frosst, N., Hinton, G.E.: Dynamic routing between capsules. In: Advances in Neural Information Processing Systems, pp. 3856–3866 (2017)
Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 618–626 (2017)
Shen, S., Han, S.X., Aberle, D.R., Bui, A.A., Hsu, W.: An interpretable deep hierarchical semantic convolutional neural network for lung nodule malignancy classification. Expert Syst. Appl. 128, 84–95 (2019)
Shen, W., Zhou, M., Yang, F., Yang, C., Tian, J.: Multi-scale convolutional neural networks for lung nodule classification. In: Ourselin, S., Alexander, D.C., Westin, C.-F., Cardoso, M.J. (eds.) IPMI 2015. LNCS, vol. 9123, pp. 588–599. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-19992-4_46
Shen, W., et al.: Multi-crop convolutional neural networks for lung nodule malignancy suspiciousness classification. Pattern Recogn. 61, 663–673 (2017)
Shen, Y., Gao, M.: Dynamic routing on deep neural network for thoracic disease classification and sensitive area localization. In: Shi, Y., Suk, H.-I., Liu, M. (eds.) MLMI 2018. LNCS, vol. 11046, pp. 389–397. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00919-9_45
Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 818–833. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10590-1_53
Zhu, W., Liu, C., Fan, W., Xie, X.: Deeplung: deep 3D dual path nets for automated pulmonary nodule detection and classification. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 673–681. IEEE (2018)
Acknowledgments
This project is partially supported by the NIH funding: R01-CA246704 and R01-CA240639.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
LaLonde, R., Torigian, D., Bagci, U. (2020). Encoding Visual Attributes in Capsules for Explainable Medical Diagnoses. In: Martel, A.L., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. MICCAI 2020. Lecture Notes in Computer Science(), vol 12261. Springer, Cham. https://doi.org/10.1007/978-3-030-59710-8_29
Download citation
DOI: https://doi.org/10.1007/978-3-030-59710-8_29
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-59709-2
Online ISBN: 978-3-030-59710-8
eBook Packages: Computer ScienceComputer Science (R0)
