Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article

A Rank Graduation Box for SAFE AI

Published: 01 January 2025 Publication History

Abstract

The growth of Artificial Intelligence applications requires to develop risk management models that can balance opportunities with risks.
We contribute to the development of Artificial Intelligence risk models proposing a Rank Graduation Box (RGB), a set of integrated statistical metrics that can measure the “Sustainability”, “Accuracy”, “Fairness” and “Explainability” of any Artificial Intelligence application.
Our metrics are consistent with each other, as they are all derived from a common underlying statistical methodology: the Lorenz curve.
The validity of the metrics is assessed by means of their practical application to both simulated and real data. The results from the comparison of alternative machine learning models to simulated data are aligned with the generating models, in general indicating linear regression models as the most accurate, regression tree models as the most fair, Random Forest models as the most robust; and leading to model explanations similar to the true ones from the generating model.
The outcomes from the application of Random Forest models to real data show that the proposed RGB metrics are more interpretable and more consistent with the expectations, with respect to standard metrics such as AUC, RMSE and Shapley values. The evidence also shows that the RGB metrics are very general and can be applied to any machine learning method, regardless of the underlying data and model.

Highlights

A new toolkit for evaluating the trustworthiness condition of AI systems.
Our method is based on the use of the Lorenz and concordance curves.
The metrics provide a measure of the compliance AI applications as a percentage.
The application shows the suitability of the approach for assessing AI methods.

References

[1]
Efron B., Stein C., The jackknife estimate of variance, The Annals of Statistics 9 (3) (1981) 586–596,.
[2]
Ferrari P., Raffinetti E., A different approach to dependence analysis, Multivariate Behavioral Research 50 (2) (2015) 248–264,.
[3]
Giudici P., Centurelli M., Turchetta S., Artificial intelligence risk measurement, Expert Systems with Applications 235 (2024),.
[4]
Giudici P., Raffinetti E., Shapley-lorenz explainable artificial intelligence, Expert Systems With Applications 167 (2021),.
[5]
Giudici P., Raffinetti E., SAFE artificial intelligence in finance, Finance Research Letters 13 (2023),.
[6]
Giudici P., Raffinetti E., RGA: a unified approach of predictive accuracy, Advances in Data Analysis and Applications (2024),. (in press).
[7]
Gneiting T., Making and evaluating point forecasts, Journal of the American Statistical Association 106 (494) (2011) 746–762,.
[8]
Hand D., Till R., A simple generalisation of the area Under the ROC curve for multiple class classification problem, Machine Learning 45 (2001) 171–186,.
[9]
Hoeffding W., A class of statistcs with asymptotically normal distribution, The Annals of Mathematical Statistics 19 (3) (1948) 293–325,.
[10]
Lorenz M., Methods of measuring the concentration wealth, Publications of the American Statistical Association 9 (70) (1905) 209–219,.
[11]
Lundberg, S., & Lee, S. (2017). A Unified Approach to Interpreting Model Predictions. In NIPS’17: Proceedings of the 31st international conference on neural information processing systems (pp. 4768–4777).
[12]
Nair S., et al., Maximum likelihood uncertainty estimation: Robustness to outliers, 2022, See URL https://arxiv.org/abs/2202.03870.
[13]
Quy L., et al., A survey on datasets for fairness-aware machine learning, Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 12 (3) (2022),.
[14]
Raffinetti E., A rank graduation accuracy measure to mitigate artificial intellgence risks, Quality and Quantity 57 (2) (2023) 131–150,.
[15]
Shapley L., A value for n-person games, in: Kuhn H., Tucker A. (Eds.), Contributions to the theory of games II, Princeton University Press, 1953.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image Expert Systems with Applications: An International Journal
Expert Systems with Applications: An International Journal  Volume 259, Issue C
Jan 2025
1577 pages

Publisher

Pergamon Press, Inc.

United States

Publication History

Published: 01 January 2025

Author Tags

  1. Concordance
  2. Lorenz curve
  3. Machine learning
  4. Ranks
  5. Risk management

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 0
    Total Downloads
  • Downloads (Last 12 months)0
  • Downloads (Last 6 weeks)0
Reflects downloads up to 04 Feb 2025

Other Metrics

Citations

View Options

View options

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media