Abstract
Reliability estimation of Machine Learning (ML) models is becoming a crucial subject. This is particularly the case when such models are deployed in safety-critical applications, as the decisions based on model predictions can result in hazardous situations. In this regard, recent research has proposed methods to achieve safe, dependable, and reliable ML systems. One such method consists of detecting and analyzing distributional shift, and then measuring how such systems respond to these shifts. This was proposed in earlier work in SafeML. This work focuses on the use of SafeML for time series data, and on reliability and robustness estimation of ML-forecasting methods using statistical distance measures. To this end, distance measures based on the Empirical Cumulative Distribution Function (ECDF) proposed in SafeML are explored to measure Statistical-Distance Dissimilarity (SDD) across time series. We then propose SDD-based Reliability Estimate (StaDRe) and SDD-based Robustness (StaDRo) measures. With the help of a clustering technique, the similarity between the statistical properties of data seen during training and the forecasts is identified. The proposed method is capable of providing a link between dataset SDD and Key Performance Indicators (KPIs) of the ML models.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Adomavicius, G., Wang, Y.: Improving reliability estimation for individual numeric predictions: a machine learning approach. INFORMS J. Comput. 34, 1–669 (2021)
Anh, D.T., Thanh, L.H.: An efficient implementation of k-means clustering for time series data with DTW distance. Int. J. Bus. Intell. Data Min. 10(3), 213–232 (2015)
Aslansefat, K., Kabir, S., Abdullatif, A., Vasudevan, V., Papadopoulos, Y.: Toward improving confidence in autonomous vehicle software: a study on traffic sign recognition systems. Computer 54(8), 66–76 (2021)
Aslansefat, K., Sorokos, I., Whiting, D., Tavakoli Kolagari, R., Papadopoulos, Y.: SafeML: safety monitoring of machine learning classifiers through statistical difference measures. In: Zeller, M., Höfig, K. (eds.) IMBSA 2020. LNCS, vol. 12297, pp. 197–211. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58920-2_13
Avizienis, A., Laprie, J.C., Randell, B., Landwehr, C.: Basic concepts and taxonomy of dependable and secure computing. IEEE Trans. Dependable Secure Comput. 1(1), 11–33 (2004)
Becker, A., Becker, J.: Dataset shift assessment measures in monitoring predictive models. Procedia Comput. Sci. 192, 3391–3402 (2021). https://doi.org/10.1016/j.procs.2021.09.112. https://www.sciencedirect.com/science/article/pii/S1877050921018512. Knowledge-Based and Intelligent Information & Engineering Systems: Proceedings of the 25th International Conference KES2021
Bellemare, M.G., et al.: The Cramer distance as a solution to biased Wasserstein gradients. arXiv preprint arXiv:1705.10743 (2017)
Bosnić, Z., Kononenko, I.: Estimation of individual prediction reliability using the local sensitivity analysis. Appl. Intell. 29(3), 187–203 (2008)
Bosnić, Z., Kononenko, I.: An overview of advances in reliability estimation of individual predictions in machine learning. Intell. Data Anal. 13(2), 385–401 (2009)
Bradley, P.S., Fayyad, U.M.: Refining initial points for K-Means clustering. In: Proceedings 15th International Conference on Machine Learning, pp. 91–99. Morgan Kaufmann, San Francisco (1998)
Briesemeister, S., Rahnenführer, J., Kohlbacher, O.: No longer confidential: estimating the confidence of individual regression predictions. PLoS ONE 7(11), e48723 (2012)
Cho, K., van Merriënboer, B., Bahdanau, D., Bengio, Y.: On the properties of neural machine translation: encoder–decoder approaches. In: Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pp. 103–111. Association for Computational Linguistics, Doha, October 2014. https://doi.org/10.3115/v1/W14-4012. https://aclanthology.org/W14-4012
Demut, I.R.: Reliability of predictions in regression models. Doktorandske dny 2010 (2010)
Driemel, A., Krivošija, A., Sohler, C.: Clustering time series under the Fréchet distance. In: Proceedings of the Twenty-Seventh Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 766–785. SIAM (2016)
Efron, B.: Bootstrap methods: another look at the jackknife. In: Kotz, S., Johnson, N.L. (eds.) Breakthroughs in Statistics, pp. 569–593. Springer, New York (1992). https://doi.org/10.1007/978-1-4612-4380-9_41
Gammerman, A., Vovk, V., Vapnik, V.: Learning by transduction. In: Cooper, G.F., Moral, S. (eds.) UAI 1998: Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, University of Wisconsin Business School, Madison, Wisconsin, USA, 24–26 July 1998, pp. 148–155. Morgan Kaufmann (1998). https://dslpitt.org/uai/displayArticleDetails.jsp?mmnu=1 &smnu=2 &article_id=243 &proceeding_id=14
Hendrycks, D., Mu, N., Cubuk, E.D., Zoph, B., Gilmer, J., Lakshminarayanan, B.: AugMix: a simple data processing method to improve robustness and uncertainty. In: Proceedings of the International Conference on Learning Representations (ICLR) (2020)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997). https://doi.org/10.1162/neco.1997.9.8.1735
Hond, D., Asgari, H., Jeffery, D., Newman, M.: An integrated process for verifying deep learning classifiers using dataset dissimilarity measures. Int. J. Artif. Intell. Mach. Learn. (IJAIML) 11(2), 1–21 (2021)
IEEE: Standard glossary of software engineering terminology. IEEE Std 610.12-1990, pp. 1–84 (1990). https://doi.org/10.1109/IEEESTD.1990.101064
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. CoRR abs/1412.6980 (2015)
Lee, T., Xiao, Y., Meng, X., Duling, D.: Clustering time series based on forecast distributions using Kullback-Leibler divergence. In: International Institute of Forecasters (IIF). Web (2014)
Moreno-Torres, J.G., Raeder, T., Alaiz-Rodríguez, R., Chawla, N.V., Herrera, F.: A unifying view on dataset shift in classification. Pattern Recogn. 45(1), 521–530 (2012)
Nouretdinov, I., Melluish, T., Vovk, V.: Ridge regression confidence machine. In: Brodley, C.E., Danyluk, A.P. (eds.) Proceedings of the Eighteenth International Conference on Machine Learning, pp. 385–392. Morgan Kaufmann, San Francisco (2001)
Oliveira, G.H., Cavalcante, R.C., Cabral, G.G., Minku, L.L., Oliveira, A.L.: Time series forecasting in the presence of concept drift: a PSO-based approach. In: 2017 IEEE 29th International Conference on Tools with Artificial Intelligence (ICTAI), pp. 239–246. IEEE (2017)
Rabanser, S., Günnemann, S., Lipton, Z.: Failing loudly: an empirical study of methods for detecting dataset shift. In: Advances in Neural Information Processing Systems, vol. 32 (2019)
Ratanamahatana, C.A., Keogh, E.: Everything you know about dynamic time warping is wrong. In: Third Workshop on Mining Temporal and Sequential Data, vol. 32. Citeseer (2004)
Rauber, J., Zimmermann, R., Bethge, M., Brendel, W.: Foolbox native: fast adversarial attacks to benchmark the robustness of machine learning models in PyTorch, TensorFlow, and JAX. J. Open Source Softw. 5(53), 2607 (2020). https://doi.org/10.21105/joss.02607
Raza, H., Prasad, G., Li, Y.: EWMA model based shift-detection methods for detecting covariate shifts in non-stationary environments. Pattern Recogn. 48(3), 659–669 (2015)
Sakoe, H., Chiba, S.: Dynamic programming algorithm optimization for spoken word recognition. IEEE Trans. Acoust. Speech Sig. Process. 26(1), 43–49 (1978). https://doi.org/10.1109/TASSP.1978.1163055
Santurkar, S., Tsipras, D., Madry, A.: BREEDS: benchmarks for subpopulation shift. arXiv preprint arXiv:2008.04859 (2020)
Storkey, A.: When training and test sets are different: characterizing learning transfer. In: Dataset Shift in Machine Learning, vol. 30, pp. 3–28 (2009)
Taori, R., Dave, A., Shankar, V., Carlini, N., Recht, B., Schmidt, L.: Measuring robustness to natural distribution shifts in image classification. Adv. Neural. Inf. Process. Syst. 33, 18583–18599 (2020)
Wang, B., Webb, S., Rainforth, T.: Statistically robust neural network classification. In: de Campos, C., Maathuis, M.H. (eds.) Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence. Proceedings of Machine Learning Research, vol. 161, pp. 1735–1745. PMLR, 27–30 July 2021. https://proceedings.mlr.press/v161/wang21b.html
Wang, Y., Ma, X., Bailey, J., Yi, J., Zhou, B., Gu, Q.: On the convergence and robustness of adversarial training. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 97, pp. 6586–6595. PMLR, 09–15 June 2019. https://proceedings.mlr.press/v97/wang19i.html
Webb, S., Rainforth, T., Teh, Y.W., Kumar, M.P.: A statistical approach to assessing neural network robustness. In: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, 6–9 May 2019. OpenReview.net (2019). https://openreview.net/forum?id=S1xcx3C5FX
Zhang, H., Yu, Y., Jiao, J., Xing, E., Ghaoui, L.E., Jordan, M.: Theoretically principled trade-off between robustness and accuracy. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 97, pp. 7472–7482. PMLR, 09–15 June 2019. https://proceedings.mlr.press/v97/zhang19p.html
Zhao, X., et al.: Assessing the reliability of deep learning classifiers through robustness evaluation and operational profiles. In: AISafety 2021 Workshop at IJCAI 2021, vol. 2916 (2021). ceur-ws.org
Acknowledgments
This work was supported by the Building Trust in Ecosystems and Ecosystem Component (BIECO) Horizon 2020 project under grant agreement no. 952702.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Akram, M.N., Ambekar, A., Sorokos, I., Aslansefat, K., Schneider, D. (2022). StaDRe and StaDRo: Reliability and Robustness Estimation of ML-Based Forecasting Using Statistical Distance Measures. In: Trapp, M., Schoitsch, E., Guiochet, J., Bitsch, F. (eds) Computer Safety, Reliability, and Security. SAFECOMP 2022 Workshops . SAFECOMP 2022. Lecture Notes in Computer Science, vol 13415. Springer, Cham. https://doi.org/10.1007/978-3-031-14862-0_21
Download citation
DOI: https://doi.org/10.1007/978-3-031-14862-0_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-14861-3
Online ISBN: 978-3-031-14862-0
eBook Packages: Computer ScienceComputer Science (R0)