Abstract
Artificial intelligence systems are becoming ubiquitous in everyday life as well as in high-risk environments, such as autonomous driving, medical treatment, and medicine. The opaque nature of the deep neural network raises concerns about its adoption in high-risk environments. It is important for researchers to explain how these models reach their decisions. Most of the existing methods rely on softmax to explain model decisions. However, softmax is shown to be often misleading, particularly giving unjustified high confidence even for samples far from the training data. To overcome this shortcoming, we propose Bayesian model uncertainty for producing counterfactual explanations. In this paper, we compare the counterfactual explanation of models based on Bayesian uncertainty and softmax score. This work predictively produces minimal important features, which maximally change classifier output to explain the decision-making process of the Bayesian model. We used MNIST and Caltech Bird 2011 datasets for experiments. The results show that the Bayesian model outperforms the softmax model and produces more concise and human-understandable counterfactuals.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Adiwardana D, Luong M-T, So DR, Hall J, Fiedel N, Thoppilan R, Yang Z, Kulshreshtha A, Nemade G, Lu Y, et al (2020) Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977
Justesen N, Bontrager P, Togelius J, Risi S (2019) Deep learning for video game playing. IEEE Trans Games 12(1):1–20
Ramos S, Gehrig S, Pinggera P, Franke U, Rother C (2017) Detecting unexpected obstacles for self-driving cars: fusing deep learning and geometric modeling. In: Proceedings of the 2017 IEEE intelligent vehicles symposium (IV), pp 1025–1032, IEEE
Addo PM, Guegan D, Hassani B (2018) Credit risk analysis using machine and deep learning models. Risks 6(2):38
Zia T, Ghafoor M, Tariq SA, Taj IA (2019) Robust fingerprint classification with bayesian convolutional networks. IET Image Proc 13(8):1280–1288
Jiang F, Jiang Y, Zhi H, Dong Y, Li H, Ma S, Wang Y, Dong Q, Shen H, Wang Y (2017) Artificial intelligence in healthcare: past, present and future. Stroke Vasc Neurol 2:4
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25:1097–1105
Zednik C (2019) Solving the black box problem: a normative framework for explainable artificial intelligence. Philos Technol 34:1–24
Miller T (2019) Explanation in artificial intelligence: Insights from the social sciences. Artif Intell 267:1–38
Sundararajan M, Taly A, Yan Q (2017) Axiomatic attribution for deep networks. In: International conference on machine learning, pp 3319–3328, PMLR
Selvaraju A, Das A, Vedantam R, Cogswell M, Parikh D, Batra D (2016) Grad-cam: Why did you say that? arXiv preprint arXiv:1611.07450
Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional networks. European conference on computer vision. Springer, New York, pp 818–833
Smilkov D, Thorat N, Kim B, Viégas F, Wattenberg M (2017) Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825
Lipton ZC (2018) The mythos of model interpretability: in machine learning, the concept of interpretability is both important and slippery. Queue 16(3):31–57
Ribeiro MT, Singh S, Guestrin C (2016) Why should i trust you? Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp 1135–1144
Mueller ST, Veinott ES, Hoffman RR, Klein G, Alam L, Mamun T, Clancey WJ (2021) Principles of explanation in human-AI systems. arXiv preprint arXiv:2102.04972
Molnar C (2018) A guide for making black box models explainable. https://christophm.github.io/interpretable-ml-book
Gal Y, Ghahramani Z (2016) Dropout as a bayesian approximation: representing model uncertainty in deep learning. In: International conference on machine learning, pp 1050–1059, PMLR
Blundell C, Cornebise J, Kavukcuoglu K, Wierstra D (2015) Weight uncertainty in neural network. In: International conference on machine learning, pp 1613–1622, PMLR
Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A (2016) Learning deep features for discriminative localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2921–2929
Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2017) Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision, pp 618–626
Zhang J, Bargal SA, Lin Z, Brandt J, Shen X, Sclaroff S (2018) Top-down neural attention by excitation backprop. Int J Comput Vision 126(10):1084–1102
Springenberg JT, Dosovitskiy A, Brox T, Riedmiller M (2014) Striving for simplicity: the all convolutional net. arXiv preprint arXiv:1412.6806
Zintgraf LM, Cohen TS, Adel T, Welling M (2017) Visualizing deep neural network decisions: Prediction difference analysis. arXiv preprint arXiv:1702.04595
Fong RC, Vedaldi A (2017) Interpretable explanations of black boxes by meaningful perturbation. In: Proceedings of the IEEE international conference on computer vision, pp 3429–3437
Dabkowski P, Gal Y (2017) Real time image saliency for black box classifiers. arXiv preprint arXiv:1705.07857
Chang CH, Creager E, Goldenberg A, Duvenaud D (2018) Explaining image classifiers by counterfactual generation. arXiv preprint arXiv:1807.08024
Akula A, Wang S, Zhu S-C (2020) Cocox: generating conceptual and counterfactual explanations via fault-lines. Proc AAAI Conf Artif Intel 34:2594–2601
Gal Y (2015) What my deep model doesn’t know.... http://mlg.eng.cam.ac.uk/yarin/blog_3d801aa532c1ce.html. Accessed 22 Aug 2021
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
Gohar Ali, Feras Al-Obeidat, Abdallah Tubaishat, Tehseen Zia, Muhammad Ilyas and Alvaro Rocha declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Ali, G., Al-Obeidat, F., Tubaishat, A. et al. Counterfactual explanation of Bayesian model uncertainty. Neural Comput & Applic 35, 8027–8034 (2023). https://doi.org/10.1007/s00521-021-06528-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00521-021-06528-z