research-article

Open access

Perturbation Effect: A Metric to Counter Misleading Validation of Feature Attribution

Authors:

Eduardo VeasAuthors Info & Claims

CIKM '22: Proceedings of the 31st ACM International Conference on Information & Knowledge Management

Pages 1798 - 1807

https://doi.org/10.1145/3511808.3557418

Published: 17 October 2022 Publication History

Abstract

This paper provides evidence indicating that the most commonly used metric for validating feature attribution methods in eXplainable AI (XAI) is misleading when applied to time series data. To evaluate whether an XAI method attributes importance to relevant features, these are systematically perturbed while measuring the impact on the performance of the classifier. The assumption is that a drastic performance reduction with increasing perturbation of relevant features indicates that these are indeed relevant. We demonstrate empirically that this assumption is incomplete without considering low relevance features in the used metrics. We introduce a novel metric, the Perturbation Effect Size, and demonstrate how it complements existing metrics to offer a more faithful assessment of importance attribution. Finally, we contribute a comprehensive evaluation of attribution methods on time series data, considering the influence of perturbation methods and region size selection.

References

[1]

Amina Adadi and Mohammed Berrada. 2018. Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI). IEEE Access, Vol. 6 (2018), 52138--52160.

[2]

Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. 2018. Sanity Checks for Saliency Maps. In Advances in Neural Information Processing Systems, S Bengio, H Wallach, H Larochelle, K Grauman, N Cesa-Bianchi, and R Garnett (Eds.), Vol. 31. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2018/file/294a8ed24b1ad22ec2e7efea049b8737-Paper.pdf

[3]

David Alvarez-Melis and Tommi S Jaakkola. 2018. Towards robust interpretability with self-explaining neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems. 7786--7795.

[4]

Julia Amann, Alessandro Blasimme, Effy Vayena, Dietmar Frey, and Vince I Madai. 2020. Explainability for artificial intelligence in healthcare: a multidisciplinary perspective. BMC Medical Informatics and Decision Making, Vol. 20, 1 (2020), 1--9.

[5]

L. Arras, Grégoire Montavon, K. Müller, and W. Samek. 2017. Explaining Recurrent Neural Network Predictions in Sentiment Analysis. In WASSA@EMNLP.

[6]

Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. 2015. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, Vol. 10, 7 (2015), e0130140.

[7]

Jo ao Bento, Pedro Saleiro, André F Cruz, Mário AT Figueiredo, and Pedro Bizarro. 2021. TimeSHAP: Explaining recurrent models through sequence perturbations. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2565--2573.

Digital Library

[8]

James Bergstra, Daniel Yamins, and David Cox. 2013. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In International conference on machine learning. PMLR, 115--123.

[9]

Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. 2018. Learning to explain: An information-theoretic perspective on model interpretation. In International Conference on Machine Learning. PMLR, 883--892.

[10]

Yanping Chen, Eamonn Keogh, Bing Hu, Nurjahan Begum, Anthony Bagnall, Abdullah Mueen, and Gustavo Batista. 2015. The UCR Time Series Classification Archive. www.cs.ucr.edu/ eamonn/time_series_data/.

[11]

Ruth C Fong and Andrea Vedaldi. 2017. Interpretable explanations of black boxes by meaningful perturbation. In Proceedings of the IEEE International Conference on Computer Vision. 3429--3437.

[12]

Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018. A survey of methods for explaining black box models. ACM computing surveys (CSUR), Vol. 51, 5 (2018), 1--42.

[13]

Fred Hohman, Minsuk Kahng, Robert Pienta, and Duen Horng Chau. 2018. Visual analytics in deep learning: An interrogative survey for the next frontiers. IEEE transactions on visualization and computer graphics, Vol. 25, 8 (2018), 2674--2693.

Digital Library

[14]

Sara Hooker, Dumitru Erhan, Pieter Jan Kindermans, and Been Kim. 2019. A benchmark for interpretability methods in deep neural networks. In Advances in Neural Information Processing Systems, Vol. 32. arxiv: 1806.10758

[15]

Aya Abdelsalam Ismail, Mohamed Gunady, Hector Corrada Bravo, and Soheil Feizi. 2020. Benchmarking Deep Learning Interpretability in Time Series Predictions. In Advances in Neural Information Processing Systems, H Larochelle, M Ranzato, R Hadsell, M F Balcan, and H Lin (Eds.), Vol. 33. Curran Associates, Inc., 6441--6452. https://proceedings.neurips.cc/paper/2020/file/47a3893cc405396a5c30d91320572d6d-Paper.pdf

[16]

Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alain Muller. 2019. Deep learning for time series classification: a review. Data Mining and Knowledge Discovery, Vol. 33, 4 (01 Jul 2019), 917--963. https://doi.org/10.1007/s10618-019-00619-1

Digital Library

[17]

Hassan Ismail Fawaz, Benjamin Lucas, Germain Forestier, Charlotte Pelletier, Daniel F. Schmidt, Jonathan Weber, Geoffrey I. Webb, Lhassane Idoumghar, Pierre-Alain Muller, and François Petitjean. 2020. InceptionTime: Finding AlexNet for time series classification. Data Mining and Knowledge Discovery, Vol. 34, 6 (01 Nov 2020), 1936--1962. https://doi.org/10.1007/s10618-020-00710-y

Digital Library

[18]

Dave S Kerby. 2014. The simple difference formula: An approach to teaching nonparametric correlation. Comprehensive Psychology, Vol. 3 (2014), 11--IT.

[19]

Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. 2019. The (Un)reliability of Saliency Methods. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, Wojciech Samek, Gré goire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Müller (Eds.). Springer International Publishing, Cham, 267--280. https://doi.org/10.1007/978-3-030-28954-6_14

Digital Library

[20]

Pieter-Jan Kindermans, Kristof T Schütt, Maximilian Alber, Klaus-Robert Mü ller, Dumitru Erhan, Been Kim, and Sven D"a hne. 2018. Learning how to explain neural networks: PatternNet and PatternAttribution. In International Conference on Learning Representations. https://openreview.net/forum?id=Hkn7CBaTW

[21]

Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz-Richardson. 2020. Captum: A unified and generic model interpretability library for PyTorch. arxiv: 2009.07896 [cs.LG]

[22]

Octavio Loyola-Gonzalez. 2019. Black-box vs. white-box: Understanding their advantages and weaknesses from a practical point of view. IEEE Access, Vol. 7 (2019), 154096--154113.

[23]

Scott M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. Advances in Neural Information Processing Systems, Vol. 30 (2017), 4765--4774.

Digital Library

[24]

Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why should i trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. 1135--1144.

[25]

Cynthia Rudin. 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, Vol. 1, 5 (2019), 206--215.

[26]

Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller. 2016. Evaluating the visualization of what a deep neural network has learned. IEEE transactions on neural networks and learning systems, Vol. 28, 11 (2016), 2660--2673.

[27]

Wojciech Samek, Grégoire Montavon, Sebastian Lapuschkin, Christopher J Anders, and Klaus-Robert Müller. 2021. Explaining deep neural networks and beyond: A review of methods and applications. Proc. IEEE, Vol. 109, 3 (2021), 247--278.

[28]

Udo Schlegel, Hiba Arnout, Mennatallah El-Assady, D Oelke, and D Keim. 2019. Towards A Rigorous Evaluation Of XAI Methods On Time Series. 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) (2019), 4197--4201.

[29]

Karl Schulz, Leon Sixt, Federico Tombari, and Tim Landgraf. 2020. Restricting the Flow: Information Bottlenecks for Attribution. In International Conference on Learning Representations. https://openreview.net/forum?id=S1xWh1rYwB

[30]

Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-CAM: Visual Explanations From Deep Networks via Gradient-Based Localization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).

[31]

Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017. Learning important features through propagating activation differences. In International Conference on Machine Learning. PMLR, 3145--3153.

[32]

Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. 2016. Not Just a Black Box: Learning Important Features Through Propagating Activation Differences. CoRR, Vol. abs/1605.0 (2016). arxiv: 1605.01713 http://arxiv.org/abs/1605.01713

[33]

K Simonyan, A Vedaldi, and Andrew Zisserman. 2014. Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps. CoRR, Vol. abs/1312.6 (2014).

[34]

Akshay Sood and Mark Craven. 2021. Feature Importance Explanations for Temporal Black-Box Models. arXiv preprint arXiv:2102.11934 (2021).

[35]

Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In International Conference on Machine Learning. PMLR, 3319--3328.

[36]

Sana Tonekaboni, Shalmali Joshi, Kieran Campbell, David K Duvenaud, and Anna Goldenberg. 2020. What went wrong and when? Instance-wise feature importance for time-series black-box models. Advances in Neural Information Processing Systems, Vol. 33 (2020).

[37]

Matthew D Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In European conference on computer vision. Springer, 818--833.

[38]

Ilija Šimić, Vedran Sabol, and Eduardo Veas. 2021. XAI Methods for Neural Time Series Classification: A Brief Review. arxiv: 2108.08009 [cs.LG]

Cited By

Schlegel UKeim D(2025)Introducing the Attribution Stability Indicator: A Measure for Time Series XAI AttributionsMachine Learning and Principles and Practice of Knowledge Discovery in Databases10.1007/978-3-031-74633-8_1(3-18)Online publication date: 1-Jan-2025
https://doi.org/10.1007/978-3-031-74633-8_1
Šimić ISingh SPartl CVeas ESabol V(2024)XAIVIER: Time Series Classifier Verification with Faithful Explainable AICompanion Proceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640544.3645217(33-36)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640544.3645217
Kadir MMosavi ASonntag D(2023)Evaluation Metrics for XAI: A Review, Taxonomy, and Practical Applications2023 IEEE 27th International Conference on Intelligent Engineering Systems (INES)10.1109/INES59282.2023.10297629(000111-000124)Online publication date: 26-Jul-2023
https://doi.org/10.1109/INES59282.2023.10297629
Show More Cited By

Index Terms

Perturbation Effect: A Metric to Counter Misleading Validation of Feature Attribution
1. Computer systems organization
  1. Architectures
    1. Other architectures
      1. Neural networks

Recommendations

From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI
The rising popularity of explainable artificial intelligence (XAI) to understand high-performing black boxes raised the question of how to evaluate explanations of machine learning (ML) models. While interpretability and explainability are often presented ...
Navigating the metric maze: a taxonomy of evaluation metrics for anomaly detection in time series
Abstract
The field of time series anomaly detection is constantly advancing, with several methods available, making it a challenge to determine the most appropriate method for a specific domain. The evaluation of these methods is facilitated by the use of ...
Generating Perturbation-based Explanations with Robustness to Out-of-Distribution Data
WWW '22: Proceedings of the ACM Web Conference 2022

Perturbation-based techniques are promising for explaining black-box machine learning models due to their effectiveness and ease of implementation. However, prior works have faced the problem of Out-of-Distribution (OoD) — an artifact of randomly ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CIKM '22: Proceedings of the 31st ACM International Conference on Information & Knowledge Management

October 2022

5274 pages

ISBN:9781450392365

DOI:10.1145/3511808

General Chairs:
Mohammad Al Hasan
Indiana University Purdue University, Indianapolis, USA
,
Li Xiong
Emory University, Atlanta, USA

Copyright © 2022 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2022

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Austrian Research Promotion Agency (FFG)

Conference

CIKM '22

Sponsor:

CIKM '22: The 31st ACM International Conference on Information and Knowledge Management

October 17 - 21, 2022

GA, Atlanta, USA

Acceptance Rates

CIKM '22 Paper Acceptance Rate 621 of 2,257 submissions, 28%;

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Sponsor:
sigir
sigir

The 34th ACM International Conference on Information and Knowledge Management

November 10 - 14, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
518
Total Downloads

Downloads (Last 12 months)214
Downloads (Last 6 weeks)45

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Schlegel UKeim D(2025)Introducing the Attribution Stability Indicator: A Measure for Time Series XAI AttributionsMachine Learning and Principles and Practice of Knowledge Discovery in Databases10.1007/978-3-031-74633-8_1(3-18)Online publication date: 1-Jan-2025
https://doi.org/10.1007/978-3-031-74633-8_1
Šimić ISingh SPartl CVeas ESabol V(2024)XAIVIER: Time Series Classifier Verification with Faithful Explainable AICompanion Proceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640544.3645217(33-36)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640544.3645217
Kadir MMosavi ASonntag D(2023)Evaluation Metrics for XAI: A Review, Taxonomy, and Practical Applications2023 IEEE 27th International Conference on Intelligent Engineering Systems (INES)10.1109/INES59282.2023.10297629(000111-000124)Online publication date: 26-Jul-2023
https://doi.org/10.1109/INES59282.2023.10297629
Schlegel UKeim D(2023)A Deep Dive into Perturbations as Evaluation Technique for Time Series XAIExplainable Artificial Intelligence10.1007/978-3-031-44070-0_9(165-180)Online publication date: 21-Oct-2023
https://doi.org/10.1007/978-3-031-44070-0_9

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten