POSTER: ML-Compass: A Comprehensive Assessment Framework for Machine Learning Models
Pages 1031 - 1033
Abstract
Machine learning models have made significant breakthroughs across various domains. However, it is crucial to assess these models to obtain a complete understanding of their capabilities and limitations and ensure their effectiveness and reliability in solving real-world problems. In this paper, we present a framework, termed ML-Compass, that covers a broad range of machine learning abilities, including utility evaluation, neuron analysis, robustness evaluation, and interpretability examination. We use this framework to assess seven state-of-the-art classification models on four benchmark image datasets. Our results indicate that different models exhibit significant variation, even when trained on the same dataset. This highlights the importance of using the assessment framework to comprehend their behavior.
References
[1]
Francesco Croce and Matthias Hein. 2020. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning. PMLR, 2206–2216.
[2]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition. Ieee, 248–255.
[3]
Mengnan Du, Ninghao Liu, and Xia Hu. 2019. Techniques for interpretable machine learning. Commun. ACM 63, 1 (2019), 68–77.
[4]
Li Fei-Fei, Rob Fergus, and Pietro Perona. 2004. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop. IEEE, 178–178.
[5]
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014).
[6]
Jun Guo, Wei Bao, Jiakai Wang, Yuqing Ma, Xinghai Gao, Gang Xiao, Aishan Liu, Jian Dong, Xianglong Liu, and Wenjun Wu. 2023. A comprehensive evaluation framework for deep model robustness. Pattern Recognition (2023), 109308.
[7]
Zhengyu He. 2020. Deep learning in image classification: A survey report. In 2020 2nd International Conference on Information Technology and Computer Application (ITCA). IEEE, 174–177.
[8]
Artur Jordao and Hélio Pedrini. 2021. On the effect of pruning on adversarial robustness. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 1–11.
[9]
Alex Krizhevsky, Geoffrey Hinton, 2009. Learning multiple layers of features from tiny images. (2009).
[10]
Changliu Liu, Tomer Arnon, Christopher Lazarus, Christopher Strong, Clark Barrett, Mykel J Kochenderfer, 2021. Algorithms for verifying deep neural networks. Foundations and Trends® in Optimization 4, 3-4 (2021), 244–404.
[11]
Yugeng Liu, Rui Wen, Xinlei He, Ahmed Salem, Zhikun Zhang, Michael Backes, Emiliano De Cristofaro, Mario Fritz, and Yang Zhang. 2022. { ML-Doctor} : Holistic Risk Assessment of Inference Attacks Against Machine Learning Models. In 31st USENIX Security Symposium (USENIX Security 22). 4525–4542.
[12]
Lei Ma, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Felix Juefei-Xu, Chao Xie, Li Li, Yang Liu, Jianjun Zhao, 2018. Deepmutation: Mutation testing of deep learning systems. In 2018 IEEE 29th international symposium on software reliability engineering (ISSRE). IEEE, 100–111.
[13]
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 (2017).
[14]
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. 2016. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2574–2582.
[15]
Maura Pintor, Fabio Roli, Wieland Brendel, and Battista Biggio. 2021. Fast minimum-norm adversarial attacks through adaptive norm constraints. Advances in Neural Information Processing Systems 34 (2021), 20052–20062.
[16]
Nenad Tomašev, Julien Cornebise, Frank Hutter, Shakir Mohamed, Angela Picciariello, Bec Connelly, Danielle CM Belgrave, Daphne Ezer, Fanny Cachat van der Haert, Frank Mugisha, 2020. AI for social good: unlocking the opportunity for positive impact. Nature Communications 11, 1 (2020), 2468.
[17]
Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. 2021. Neural Pruning via Growing Regularization. In International Conference on Learning Representations (ICLR).
Index Terms
- POSTER: ML-Compass: A Comprehensive Assessment Framework for Machine Learning Models
Recommendations
Interpretable machine learning assessment
AbstractWith the surge of machine learning in AI and data science, there remains an urgent need to not only compare the performance of different methods across diverse datasets but also to analyze machine learning behaviors with sensitivity using an ...
Machine Learning: The State of the Art
The two fundamental problems in machine learning (ML) are statistical analysis and algorithm design. The former tells us the principles of the mathematical models that we establish from the observation data. The latter defines the conditions on which ...
ComplAI: Framework for Multi-factor Assessment of Black-Box Supervised Machine Learning Models
SAC '23: Proceedings of the 38th ACM/SIGAPP Symposium on Applied ComputingIn this paper, we present ComplAI, a unique framework to enable, observe, analyze and quantify explainability, robustness, performance, fairness, and model's behavior in drift scenarios, and to provide a single Trust Factor that evaluates different ...
Comments
Information & Contributors
Information
Published In
July 2023
1066 pages
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 10 July 2023
Check for updates
Author Tags
Qualifiers
- Short-paper
- Research
- Refereed limited
Conference
ASIA CCS '23
Sponsor:
ASIA CCS '23: ACM Asia Conference on Computer and Communications Security
July 10 - 14, 2023
VIC, Melbourne, Australia
Acceptance Rates
Overall Acceptance Rate 418 of 2,322 submissions, 18%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 115Total Downloads
- Downloads (Last 12 months)45
- Downloads (Last 6 weeks)2
Reflects downloads up to 03 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format