Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
SpringerLink
Log in

StaDRe and StaDRo: Reliability and Robustness Estimation of ML-Based Forecasting Using Statistical Distance Measures

  • Conference paper
  • First Online:
Computer Safety, Reliability, and Security. SAFECOMP 2022 Workshops (SAFECOMP 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13415))

Included in the following conference series:

Abstract

Reliability estimation of Machine Learning (ML) models is becoming a crucial subject. This is particularly the case when such models are deployed in safety-critical applications, as the decisions based on model predictions can result in hazardous situations. In this regard, recent research has proposed methods to achieve safe, dependable, and reliable ML systems. One such method consists of detecting and analyzing distributional shift, and then measuring how such systems respond to these shifts. This was proposed in earlier work in SafeML. This work focuses on the use of SafeML for time series data, and on reliability and robustness estimation of ML-forecasting methods using statistical distance measures. To this end, distance measures based on the Empirical Cumulative Distribution Function (ECDF) proposed in SafeML are explored to measure Statistical-Distance Dissimilarity (SDD) across time series. We then propose SDD-based Reliability Estimate (StaDRe) and SDD-based Robustness (StaDRo) measures. With the help of a clustering technique, the similarity between the statistical properties of data seen during training and the forecasts is identified. The proposed method is capable of providing a link between dataset SDD and Key Performance Indicators (KPIs) of the ML models.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    www.carla.org.

  2. 2.

    https://github.com/n-akram/TimeSeriesSafeML.

References

  1. Adomavicius, G., Wang, Y.: Improving reliability estimation for individual numeric predictions: a machine learning approach. INFORMS J. Comput. 34, 1–669 (2021)

    Google Scholar 

  2. Anh, D.T., Thanh, L.H.: An efficient implementation of k-means clustering for time series data with DTW distance. Int. J. Bus. Intell. Data Min. 10(3), 213–232 (2015)

    Google Scholar 

  3. Aslansefat, K., Kabir, S., Abdullatif, A., Vasudevan, V., Papadopoulos, Y.: Toward improving confidence in autonomous vehicle software: a study on traffic sign recognition systems. Computer 54(8), 66–76 (2021)

    Article  Google Scholar 

  4. Aslansefat, K., Sorokos, I., Whiting, D., Tavakoli Kolagari, R., Papadopoulos, Y.: SafeML: safety monitoring of machine learning classifiers through statistical difference measures. In: Zeller, M., Höfig, K. (eds.) IMBSA 2020. LNCS, vol. 12297, pp. 197–211. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58920-2_13

    Chapter  Google Scholar 

  5. Avizienis, A., Laprie, J.C., Randell, B., Landwehr, C.: Basic concepts and taxonomy of dependable and secure computing. IEEE Trans. Dependable Secure Comput. 1(1), 11–33 (2004)

    Article  Google Scholar 

  6. Becker, A., Becker, J.: Dataset shift assessment measures in monitoring predictive models. Procedia Comput. Sci. 192, 3391–3402 (2021). https://doi.org/10.1016/j.procs.2021.09.112. https://www.sciencedirect.com/science/article/pii/S1877050921018512. Knowledge-Based and Intelligent Information & Engineering Systems: Proceedings of the 25th International Conference KES2021

  7. Bellemare, M.G., et al.: The Cramer distance as a solution to biased Wasserstein gradients. arXiv preprint arXiv:1705.10743 (2017)

  8. Bosnić, Z., Kononenko, I.: Estimation of individual prediction reliability using the local sensitivity analysis. Appl. Intell. 29(3), 187–203 (2008)

    Article  Google Scholar 

  9. Bosnić, Z., Kononenko, I.: An overview of advances in reliability estimation of individual predictions in machine learning. Intell. Data Anal. 13(2), 385–401 (2009)

    Article  Google Scholar 

  10. Bradley, P.S., Fayyad, U.M.: Refining initial points for K-Means clustering. In: Proceedings 15th International Conference on Machine Learning, pp. 91–99. Morgan Kaufmann, San Francisco (1998)

    Google Scholar 

  11. Briesemeister, S., Rahnenführer, J., Kohlbacher, O.: No longer confidential: estimating the confidence of individual regression predictions. PLoS ONE 7(11), e48723 (2012)

    Article  Google Scholar 

  12. Cho, K., van Merriënboer, B., Bahdanau, D., Bengio, Y.: On the properties of neural machine translation: encoder–decoder approaches. In: Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pp. 103–111. Association for Computational Linguistics, Doha, October 2014. https://doi.org/10.3115/v1/W14-4012. https://aclanthology.org/W14-4012

  13. Demut, I.R.: Reliability of predictions in regression models. Doktorandske dny 2010 (2010)

    Google Scholar 

  14. Driemel, A., Krivošija, A., Sohler, C.: Clustering time series under the Fréchet distance. In: Proceedings of the Twenty-Seventh Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 766–785. SIAM (2016)

    Google Scholar 

  15. Efron, B.: Bootstrap methods: another look at the jackknife. In: Kotz, S., Johnson, N.L. (eds.) Breakthroughs in Statistics, pp. 569–593. Springer, New York (1992). https://doi.org/10.1007/978-1-4612-4380-9_41

  16. Gammerman, A., Vovk, V., Vapnik, V.: Learning by transduction. In: Cooper, G.F., Moral, S. (eds.) UAI 1998: Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, University of Wisconsin Business School, Madison, Wisconsin, USA, 24–26 July 1998, pp. 148–155. Morgan Kaufmann (1998). https://dslpitt.org/uai/displayArticleDetails.jsp?mmnu=1 &smnu=2 &article_id=243 &proceeding_id=14

  17. Hendrycks, D., Mu, N., Cubuk, E.D., Zoph, B., Gilmer, J., Lakshminarayanan, B.: AugMix: a simple data processing method to improve robustness and uncertainty. In: Proceedings of the International Conference on Learning Representations (ICLR) (2020)

    Google Scholar 

  18. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997). https://doi.org/10.1162/neco.1997.9.8.1735

  19. Hond, D., Asgari, H., Jeffery, D., Newman, M.: An integrated process for verifying deep learning classifiers using dataset dissimilarity measures. Int. J. Artif. Intell. Mach. Learn. (IJAIML) 11(2), 1–21 (2021)

    Google Scholar 

  20. IEEE: Standard glossary of software engineering terminology. IEEE Std 610.12-1990, pp. 1–84 (1990). https://doi.org/10.1109/IEEESTD.1990.101064

  21. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. CoRR abs/1412.6980 (2015)

    Google Scholar 

  22. Lee, T., Xiao, Y., Meng, X., Duling, D.: Clustering time series based on forecast distributions using Kullback-Leibler divergence. In: International Institute of Forecasters (IIF). Web (2014)

    Google Scholar 

  23. Moreno-Torres, J.G., Raeder, T., Alaiz-Rodríguez, R., Chawla, N.V., Herrera, F.: A unifying view on dataset shift in classification. Pattern Recogn. 45(1), 521–530 (2012)

    Article  Google Scholar 

  24. Nouretdinov, I., Melluish, T., Vovk, V.: Ridge regression confidence machine. In: Brodley, C.E., Danyluk, A.P. (eds.) Proceedings of the Eighteenth International Conference on Machine Learning, pp. 385–392. Morgan Kaufmann, San Francisco (2001)

    Google Scholar 

  25. Oliveira, G.H., Cavalcante, R.C., Cabral, G.G., Minku, L.L., Oliveira, A.L.: Time series forecasting in the presence of concept drift: a PSO-based approach. In: 2017 IEEE 29th International Conference on Tools with Artificial Intelligence (ICTAI), pp. 239–246. IEEE (2017)

    Google Scholar 

  26. Rabanser, S., Günnemann, S., Lipton, Z.: Failing loudly: an empirical study of methods for detecting dataset shift. In: Advances in Neural Information Processing Systems, vol. 32 (2019)

    Google Scholar 

  27. Ratanamahatana, C.A., Keogh, E.: Everything you know about dynamic time warping is wrong. In: Third Workshop on Mining Temporal and Sequential Data, vol. 32. Citeseer (2004)

    Google Scholar 

  28. Rauber, J., Zimmermann, R., Bethge, M., Brendel, W.: Foolbox native: fast adversarial attacks to benchmark the robustness of machine learning models in PyTorch, TensorFlow, and JAX. J. Open Source Softw. 5(53), 2607 (2020). https://doi.org/10.21105/joss.02607

  29. Raza, H., Prasad, G., Li, Y.: EWMA model based shift-detection methods for detecting covariate shifts in non-stationary environments. Pattern Recogn. 48(3), 659–669 (2015)

    Article  Google Scholar 

  30. Sakoe, H., Chiba, S.: Dynamic programming algorithm optimization for spoken word recognition. IEEE Trans. Acoust. Speech Sig. Process. 26(1), 43–49 (1978). https://doi.org/10.1109/TASSP.1978.1163055

    Article  MATH  Google Scholar 

  31. Santurkar, S., Tsipras, D., Madry, A.: BREEDS: benchmarks for subpopulation shift. arXiv preprint arXiv:2008.04859 (2020)

  32. Storkey, A.: When training and test sets are different: characterizing learning transfer. In: Dataset Shift in Machine Learning, vol. 30, pp. 3–28 (2009)

    Google Scholar 

  33. Taori, R., Dave, A., Shankar, V., Carlini, N., Recht, B., Schmidt, L.: Measuring robustness to natural distribution shifts in image classification. Adv. Neural. Inf. Process. Syst. 33, 18583–18599 (2020)

    Google Scholar 

  34. Wang, B., Webb, S., Rainforth, T.: Statistically robust neural network classification. In: de Campos, C., Maathuis, M.H. (eds.) Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence. Proceedings of Machine Learning Research, vol. 161, pp. 1735–1745. PMLR, 27–30 July 2021. https://proceedings.mlr.press/v161/wang21b.html

  35. Wang, Y., Ma, X., Bailey, J., Yi, J., Zhou, B., Gu, Q.: On the convergence and robustness of adversarial training. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 97, pp. 6586–6595. PMLR, 09–15 June 2019. https://proceedings.mlr.press/v97/wang19i.html

  36. Webb, S., Rainforth, T., Teh, Y.W., Kumar, M.P.: A statistical approach to assessing neural network robustness. In: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, 6–9 May 2019. OpenReview.net (2019). https://openreview.net/forum?id=S1xcx3C5FX

  37. Zhang, H., Yu, Y., Jiao, J., Xing, E., Ghaoui, L.E., Jordan, M.: Theoretically principled trade-off between robustness and accuracy. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 97, pp. 7472–7482. PMLR, 09–15 June 2019. https://proceedings.mlr.press/v97/zhang19p.html

  38. Zhao, X., et al.: Assessing the reliability of deep learning classifiers through robustness evaluation and operational profiles. In: AISafety 2021 Workshop at IJCAI 2021, vol. 2916 (2021). ceur-ws.org

Download references

Acknowledgments

This work was supported by the Building Trust in Ecosystems and Ecosystem Component (BIECO) Horizon 2020 project under grant agreement no. 952702.

Author information

Authors and Affiliations

  1. University of Hull, Hull, UK

    Koorosh Aslansefat

  2. Fraunhofer IESE, Kaiserslautern, Germany

    Mohammed Naveed Akram, Ioannis Sorokos & Daniel Schneider

  3. Technical University of Kaiserslautern, Kaiserslautern, Germany

    Akshatha Ambekar

Authors
  1. Mohammed Naveed Akram
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Akshatha Ambekar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ioannis Sorokos
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Koorosh Aslansefat
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Daniel Schneider
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Mohammed Naveed Akram or Ioannis Sorokos .

Editor information

Editors and Affiliations

  1. Fraunhofer IKS, Munich, Germany

    Mario Trapp

  2. Austrian Institute of Technology, Vienna, Austria

    Erwin Schoitsch

  3. LAAS-CNRS, University of Toulouse, Toulouse, France

    Jérémie Guiochet

  4. Thales Deutschland GmbH, Ditzingen, Germany

    Friedemann Bitsch

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Akram, M.N., Ambekar, A., Sorokos, I., Aslansefat, K., Schneider, D. (2022). StaDRe and StaDRo: Reliability and Robustness Estimation of ML-Based Forecasting Using Statistical Distance Measures. In: Trapp, M., Schoitsch, E., Guiochet, J., Bitsch, F. (eds) Computer Safety, Reliability, and Security. SAFECOMP 2022 Workshops . SAFECOMP 2022. Lecture Notes in Computer Science, vol 13415. Springer, Cham. https://doi.org/10.1007/978-3-031-14862-0_21

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-14862-0_21

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-14861-3

  • Online ISBN: 978-3-031-14862-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation