research-article

Free access

Meaningful explanations of black box AI decision systems

AUTHORs:

Dino Pedreschi,

Fosca Giannotti,

Riccardo Guidotti,

Salvatore Ruggieri,

Franco TuriniAuthors Info & Claims

AAAI'19/IAAI'19/EAAI'19: Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence

Article No.: 1213, Pages 9780 - 9784

https://doi.org/10.1609/aaai.v33i01.33019780

Published: 27 January 2019 Publication History

PDF eReader Publisher Site

Abstract

Black box AI systems for automated decision making, often based on machine learning over (big) data, map a user's features into a class or a score without exposing the reasons why. This is problematic not only for lack of transparency, but also for possible biases inherited by the algorithms from human prejudices and collection artifacts hidden in the training data, which may lead to unfair or wrong decisions. We focus on the urgent open challenge of how to construct meaningful explanations of opaque AI/ML systems, introducing the local-to-global framework for black box explanation, articulated along three lines: (i) the language for expressing explanations in terms of logic rules, with statistical and causal interpretation; (ii) the inference of local explanations for revealing the decision rationale for a specific case, by auditing the black box in the vicinity of the target instance; (iii), the bottom-up generalization of many local explanations into simple global ones, with algorithms that optimize for quality and comprehensibility. We argue that the local-first approach opens the door to a wide variety of alternative solutions along different dimensions: a variety of data sources (relational, text, images, etc.), a variety of learning problems (multi-label classification, regression, scoring, ranking), a variety of languages for expressing meaningful explanations, a variety of means to audit a black box.

References

[1]

Andrews, R.; Diederich, J.; and Tickle, A. B. 1995. Survey and critique of techniques for extracting rules from trained artificial neural networks. Knowl.-Based Syst. 8(6):373–389.

Digital Library

[2]

Augasta, M. G., and Kathirvalavakumar, T. 2012. Reverse engineering the neural networks for rule extraction in classification problems. NPL 35(2):131–150.

Digital Library

[3]

Bareinboim, E., and Pearl, J. 2016. Causal inference and the data-fusion problem. PNAS 113(27):7345–7352.

[4]

Bonchi, F.; Giannotti, F.; Mainetto, G.; and Pedreschi, D. 1999. A classification-based methodology for planning audit strategies in fraud detection. In KDD, 175–184. ACM.

[5]

Bonchi, F.; Hajian, S.; Mishra, B.; and Ramazzotti, D. 2017. Exposing the probabilistic causal structure of discrimination. IJDSA 3(1):1–21.

[6]

Caliskan, A.; Bryson, J. J.; and Narayanan, A. 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356:183–186.

[7]

Caravagna, G.; Graudenzi, A.; Ramazzotti, D.; Sanz-Pamplona, R.; De Sano, L.; Mauri, G.; Moreno, V.; Antoniotti, M.; and Mishra, B. 2016. Algorithmic methods to infer the evolutionary trajectories in cancer progression. PNAS 113(28):E4025–E4034.

[8]

Choi, E.; Bahadori, M. T.; Schuetz, A.; Stewart, W. F.; and Sun, J. 2016. Doctor AI: Predicting clinical events via recurrent neural networks. In PMLR, 301–318.

[9]

Craven, M., and Shavlik, J. W. 1995. Extracting tree-structured representations of trained networks. In NIPS, 24–30.

[10]

Danks, D., and London, A. J. 2017. Regulating autonomous systems: Beyond standards. IEEE IS 32(1):88–91.

[11]

Fung, G.; Sandilya, S.; and Rao, R. B. 2005. Rule extraction from linear support vector machines. In KDD, 32–40. ACM.

[12]

Goodman, B., and Flaxman, S. 2016. EU regulations on algorithmic decision-making and a "right to explanation". In ICML.

[13]

Guidotti, R.; Monreale, A.; Ruggieri, S.; Pedreschi, D.; Turini, F.; and Giannotti, F. 2018a. Local rule-based explanations of black box decision systems. arXiv:1805.10820.

[14]

Guidotti, R.; Monreale, A.; Ruggieri, S.; Turini, F.; Giannotti, F.; and Pedreschi, D. 2018b. A survey of methods for explaining black box models. CSUR 51(5):93:1–93:42.

Digital Library

[15]

Henelius, A.; Puolamäki, K.; Boström, H.; Asker, L.; and Papapetrou, P. 2014. A peek into the black box: exploring classifiers by randomization. DAMI 28(5-6):1503–1529.

[16]

Huang, B.; Zhang, K.; Lin, Y.; Scholkopf, B.; and Glymour, C. 2018. Generalized score functions for causal discovery. In KDD, 1551–1560. ACM.

[17]

Johnson, A. E. W.; Pollard, T. J.; Shen, L.; Lehman, L. H.; Feng, M.; Ghassemi, M.; Moody, B.; Szolovits, P.; Celi, L. A.; and Mark, R. G. 2016. MIMIC-III, a freely accessible critical care database. Scientific Data 3:160035.

[18]

Kingston, J. K. C. 2016. Artificial intelligence and legal liability. In SGAI Conf., 269–279. Springer.

[19]

Krause, J.; Perer, A.; and Ng, K. 2016. Interacting with predictions: Visual inspection of black-box machine learning models. In CHI, 5686–5697. New York, NY, USA: ACM.

[20]

Krishnan, R.; Sivakumar, G.; and Bhattacharya, P. 1999. Extracting decision trees from trained neural networks. Pattern recognition 32(12).

[21]

Kroll, J. A.; Huey, J.; Barocas, S.; Felten, E. W.; Reidenberg, J. R.; Robinson, D. G.; and Yu, H. 2017. Accountable algorithms. U. of Penn. Law Review 165:633–705.

[22]

Lakkaraju, H.; Bach, S. H.; and Leskovec, J. 2016. Interpretable decision sets: A joint framework for description and prediction. In KDD, 1675–1684. ACM.

[23]

Leake, D. B. 1992. Evaluating explanations: A content theory. Lawrence Erlbaum Associates.

[24]

Lou, Y.; Caruana, R.; and Gehrke, J. 2012. Intelligible models for classification and regression. In KDD, 150–158. ACM.

[25]

Malgieri, G., and Comandé, G. 2017. Why a right to legibility of automated decision-making exists in the general data protection regulation. IDPL 7(4):243–265.

[26]

Malioutov, D.; Varshney, K. R.; Emad, A.; and Dash, S. 2017. Learning interpretable classification rules with boolean compressed sensing. In Transparent Data Mining for Big and Small Data. Springer. 95–121.

[27]

Nanni, M.; Trasarti, R.; Monreale, A.; Grossi, V.; and Pedreschi, D. 2016. Driving profiles computation and monitoring for car insurance CRM. ACM Trans. Intell. Syst. Technol. 8(1):14:1–14:26.

Digital Library

[28]

Nugent, C., and Cunningham, P. 2005. A case-based explanation system for black-box systems. AIR 24(2):163–178.

Digital Library

[29]

Pasquale, F. 2015. The black box society: The secret algorithms that control money and information. HUP.

[30]

Pearl, J., and MacKenzie, D. 2018. The Book of Why: the new science of cause and effect. Basic Books.

Digital Library

[31]

Pedreschi, D.; Giannotti, F.; Guidotti, R.; Monreale, A.; Pappalardo, L.; Ruggieri, S.; and Turini, F. 2018. Open the black box data-driven explanation of black box decision systems. arXiv:1806.09936.

[32]

Pedreschi, D.; Ruggieri, S.; and Turini, F. 2008. Discrimination-aware data mining. In KDD, 560. ACM.

[33]

Peters, J.; Janzing, D.; and Scholkopf, B. 2017. Elements of causal inference: foundations and learning algorithms. MIT Press.

[34]

Ribeiro, M. T.; Singh, S.; and Guestrin, C. 2016. Why should i trust you?: Explaining the predictions of any classifier. In KDD, 1135–1144. ACM.

[35]

Ribeiro, M. T.; Singh, S.; and Guestrin, C. 2018. Anchors: High-precision model-agnostic explanations. In AAAI, 1527–1535.

[36]

Ruggieri, S.; Pedreschi, D.; and Turini, F. 2010. Data mining for discrimination discovery. TKDD 4(2):9:1–9:40.

Digital Library

[37]

Singh, J., and Anand, A. 2018. Exs: Explainable search using local model agnostic interpretability. arXiv:1809.03857.

[38]

Tan, H. F.; Hooker, G.; and Wells, M. T. 2016. Tree space prototypes: Another look at making tree ensembles interpretable. arXiv:1611.07115.

[39]

Xu, K.; Ba, J.; Kiros, R.; Cho, K.; Courville, A.; Salakhutdinov, R.; Zemel, R.; and Bengio, Y. 2015. Show, attend and tell: Neural image caption generation with visual attention. In ICML, 2048.

[40]

Zhang, J., and Bareinboim, E. 2018. Fairness in decision-making: The causal explanation formula. In AAAI, 2037.

[41]

Zhou, B.; Khosla, A.; Lapedriza, A.; Oliva, A.; and Torralba, A. 2016. Learning deep features for discriminative localization. In CVPR, 2921–2929. IEEE.

Cited By

Xiao YZhang ZFang YYan DZhou YKu WHui BSerra ESpezzano F(2024)Advancing Certified Robustness of Explanation via Gradient QuantizationProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3679650(2596-2606)Online publication date: 21-Oct-2024
https://dl.acm.org/doi/10.1145/3627673.3679650
Bouazizi SLtifi H(2024)Novel diversified echo state network for improved accuracy and explainability of EEG-based stroke predictionInformation Systems10.1016/j.is.2023.102317120:COnline publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1016/j.is.2023.102317
Cau FHauptmann HSpano LTintarev N(2023)Supporting High-Uncertainty Decisions through AI and Logic-Style ExplanationsProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584080(251-263)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3581641.3584080
Show More Cited By

Index Terms

Meaningful explanations of black box AI decision systems

Index terms have been assigned to the content through auto-classification.

Recommendations

A Survey of Methods for Explaining Black Box Models

In recent years, many accurate decision support systems have been constructed as black boxes, that is as systems that hide their internal logic to the user. This lack of explanation constitutes both a practical and an ethical issue. The literature ...
Robust and stable black box explanations
ICML'20: Proceedings of the 37th International Conference on Machine Learning

As machine learning black boxes are increasingly being deployed in real-world applications, there has been a growing interest in developing post hoc explanations that summarize the behaviors of these black boxes. However, existing algorithms for ...
Considerations when learning additive explanations for black-box models
Abstract
Many methods to explain black-box models, whether local or global, are additive. In this paper, we study global additive explanations for non-additive models, focusing on four explanation methods: partial dependence, Shapley explanations adapted ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

AAAI'19/IAAI'19/EAAI'19: Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence

January 2019

10088 pages

ISBN:978-1-57735-809-1

Copyright © 2019 Association for the Advancement of Artificial Intelligence.

Sponsors

Association for the Advancement of Artificial Intelligence

Publisher

AAAI Press

Publication History

Published: 27 January 2019

Qualifiers

Research-article
Research
Refereed limited

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
104
Total Downloads

Downloads (Last 12 months)58
Downloads (Last 6 weeks)10

Reflects downloads up to 27 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Xiao YZhang ZFang YYan DZhou YKu WHui BSerra ESpezzano F(2024)Advancing Certified Robustness of Explanation via Gradient QuantizationProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3679650(2596-2606)Online publication date: 21-Oct-2024
https://dl.acm.org/doi/10.1145/3627673.3679650
Bouazizi SLtifi H(2024)Novel diversified echo state network for improved accuracy and explainability of EEG-based stroke predictionInformation Systems10.1016/j.is.2023.102317120:COnline publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1016/j.is.2023.102317
Cau FHauptmann HSpano LTintarev N(2023)Supporting High-Uncertainty Decisions through AI and Logic-Style ExplanationsProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584080(251-263)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3581641.3584080
Kim SWatkins ERussakovsky OFong RMonroy-Hernández A(2023)"Help Me Help the AI": Understanding How Explainability Can Support Human-AI InteractionProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581001(1-17)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581001
O'Neill TFlathmann CMcNeese NSalas E(2023)21st Century teaming and beyondComputers in Human Behavior10.1016/j.chb.2023.107865147:COnline publication date: 1-Oct-2023
https://dl.acm.org/doi/10.1016/j.chb.2023.107865
Messner W(2023)From black box to clear boxApplied Soft Computing10.1016/j.asoc.2023.110729146:COnline publication date: 17-Oct-2023
https://dl.acm.org/doi/10.1016/j.asoc.2023.110729
Wickstrøm KTrosten DLøkse SBoubekki AMikalsen KKampffmeyer MJenssen R(2023)RELAX: Representation Learning ExplainabilityInternational Journal of Computer Vision10.1007/s11263-023-01773-2131:6(1584-1610)Online publication date: 11-Mar-2023
https://dl.acm.org/doi/10.1007/s11263-023-01773-2
Newn JKelly RD'Alfonso SLederman R(2022)Examining and Promoting Explainable Recommendations for Personal Sensing Technology AcceptanceProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35502976:3(1-27)Online publication date: 7-Sep-2022
https://dl.acm.org/doi/10.1145/3550297
Werder KRamesh BZhang R(2022)Establishing Data Provenance for Responsible Artificial Intelligence SystemsACM Transactions on Management Information Systems10.1145/350348813:2(1-23)Online publication date: 10-Mar-2022
https://dl.acm.org/doi/10.1145/3503488
Barducci AIannaccone SLa Gatta VMoscato VSperlì GZavota S(2022)An end-to-end framework for information extraction from Italian resumesExpert Systems with Applications: An International Journal10.1016/j.eswa.2022.118487210:COnline publication date: 30-Dec-2022
https://dl.acm.org/doi/10.1016/j.eswa.2022.118487
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten