MoDALAS: addressing assurance for learning-enabled autonomous systems in the face of uncertainty
Authors: 
Michael Austin Langford, Kenneth H. Chan, Jonathon Emil Fleck, Philip K. McKinley, Betty H. C. ChengAuthors Info & Claims
Michael Austin Langford, Kenneth H. Chan, Jonathon Emil Fleck, Philip K. McKinley, Betty H. C. ChengAuthors Info & ClaimsPages 1543 - 1563
Abstract
Increasingly, safety-critical systems include artificial intelligence and machine learning components (i.e., learning-enabled components (LECs)). However, when behavior is learned in a training environment that fails to fully capture real-world phenomena, the response of an LEC to untrained phenomena is uncertain and therefore cannot be assured as safe. Automated methods are needed for self-assessment and adaptation to decide when learned behavior can be trusted. This work introduces a model-driven approach to manage self-adaptation of a learning-enabled system (LES) to account for run-time contexts for which the learned behavior of LECs cannot be trusted. The resulting framework enables an LES to monitor and evaluate goal models at run time to determine whether or not LECs can be expected to meet functional objectives and enables system adaptation accordingly. Using this framework enables stakeholders to have more confidence that LECs are used only in contexts comparable to those validated at design time.
References
[1]
Cámara J, de Lemos R, Ghezzi C, and Lopes A Assurances for Self-Adaptive Systems: Principles, Models, and Techniques 2013 Berlin, Heidelberg Springer
[2]
Langari, Z., Maibaum, T.: Safety cases: a review of challenges. Paper presented at 1st Int. Workshop on Assurance Cases for Software-Intensive Systems (ASSURE 2013) (2013)
[3]
Kocić J, Jovičić N, and Drndarević V An end-to-end deep neural network for autonomous driving designed for embedded automotive platforms Sensors 2019 19 9 2064
[4]
Wu, B., Iandola, F., Jin, P.H., Keutzer, K.: Squeezedet: unified, small, low power fully convolutional neural networks for real-time object detection for autonomous driving (2017)
[5]
Yao, X., Wang, X., Wang, S.-H., Zhang, Y.-D.: A comprehensive survey on convolutional neural network in medical image analysis. Multimed. Tools Appl. 1–45 (2020)
[6]
Abdou, M.A.: Literature review: efficient deep neural networks techniques for medical image analysis. Neural Comput. Appl. 1–22 (2022)
[7]
Lu R and Hong SH Incentive-based demand response for smart grid with reinforcement learning and deep neural network Appl. Energy 2019 236 937-949
[8]
Tuncali, C.E., Kapinski, J., Ito, H., Deshmukh, J.V.: Reasoning about safety of learning-enabled components in autonomous cyber-physical systems. Paper presented at 55th Annual Design Automation Conf. (DAC 2018) (2018)
[9]
Kawaguchi, K., Kaelbling, L.P., Bengio, Y.: Generalization in deep learning. Tech. Rep., MIT (2018). https://lis.csail.mit.edu/pubs/kawaguchi-techreport18.pdf
[10]
Yu, F., et al.: Interpreting and evaluating neural network robustness. Paper presented at 28th International Joint Conf. on Artificial Intelligence (IJCAI 2019) (2019)
[11]
Knight, W.: The Dark Secret at the Heart of AI. MIT Technology Review Artificial intelligence/Machine learning (2017). https://www.technologyreview.com/2017/04/11/5113/the-dark-secret-at-the-heart-of-ai/
[12]
Rushby, J.: The Interpretation and Evaluation of Assurance Cases. Tech. Rep. SRI-CSL-15-01, Computer Science Laboratory, SRI International, Menlo Park, CA (2015). https://www.csl.sri.com/users/rushby/papers/sri-csl-15-1-assurance-cases.pdf
[13]
2022 IEEE Conference on Assured Autonomy (ICAA) (2022)
[14]
Assured autonomy workshop series. https://cra.org/ccc/visioning/visioning-activities/2019-activities/assured-autonomy/. Accessed: 2022-08-22
[15]
Workshop on assured autonomous systems 2020 (2020)
[16]
Air Force Office of Scientific Research.: Center of excellence: Assured autonomy in contested environments. https://www.federalgrants.com/CENTER-OF-EXCELLENCE-Assured-Autonomy-in-Contested-Environments-71233.html (2018)
[17]
Neema, S.: Assured autonomy. https://www.darpa.mil/program/assured-autonomy (2017)
[18]
Schumann J, Gupta P, and Liu Y Application of Neural Networks in High Assurance Systems: A Survey of Studies in Computational Intelligence (SCI) 2010 Berlin, Heidelberg Springer
[19]
Hartsell, C., et al.: Model-based design for CPS with learning-enabled components. Paper presented at Workshop on Design Automation for CPS and IoT (DESTION 2019) (2019)
[20]
van Lamsweerde, A., Letier, E.: From object orientation to goal orientation: a paradigm shift for requirements engineering. Paper presented at Radical Innovations of Software and Systems Engineering in the Future (RISSEF 2002) (2004)
[21]
Kephart JO and Das R Achieving self-management via utility functions IEEE Internet Comput. 2007 11 1 40-48
[22]
Kephart JO and Chess DM The vision of autonomic computing Computer 2003 36 1 41-50
[23]
Langford, M.A., Cheng, B. H.C.: “Know What You Know”: predicting behavior for learning-enabled systems when facing uncertainty. Paper presented at 16th Int. Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2021) (2021)
[24]
Barr ET, Harman M, McMinn P, Shahbaz M, and Yoo S The oracle problem in software testing: a survey Trans. Softw. Eng. 2015 41 5 507-525
[25]
Whittle, J., Sawyer, P., Bencomo, N., Cheng, B.H.C.,Bruel, J.-M.: RELAX: incorporating uncertainty into the specification of self-adaptive systems. Paper presented at 17th IEEE Int. Requirements Engineering Conf. (RE 2009) (2009)
[26]
Whittle J, Sawyer P, Bencomo N, Cheng BHC, and Bruel J-M RELAX: a language to address uncertainty in self-adaptive systems requirement Requir. Eng. 2010 15 2 177-196
[27]
Quigley, M., et al.: ROS: an open-source robot operating system. Paper presented at Int. Conf. on Robotics and Automation Workshop on Open Source Software (ICRA Workshop 2009) (2009)
[28]
Jiao L et al. A survey of deep learning-based object detection IEEE Access 2019 7 128837-128868
[29]
Goodfellow I, Bengio Y, and Courville A Deep Learning 2016 Cambridge MIT
[30]
Krizhevsky A, Sutskever I, and Hinton GE ImageNet classification with deep convolutional neural networks Commun. ACM 2017 60 6 84-90
[31]
Liu L et al. Deep learning for generic object detection: a survey Int. J. Comput. Vis. 2018 128 261-318
[32]
Kuutti S, Bowden R, Jin Y, Barber P, and Fallah S A vehicle survey of deep learning applications to autonomous control IEEE Trans. Intell. Transp. Syst. 2021 22 2 712-733
[33]
Ravindran R, Santora MJ, and Jamali MM Multi-object detection and tracking, based on DNN, for autonomous vehicles: a review IEEE Sens. J. 2020 21 5 5668-5677
[34]
Schwarting W, Alonso-Mora J, and Rus D Planning and decision-making for autonomous vehicles Annu. Rev. Control Robot. Auton. Syst. 2018 1 1 187-210
[35]
Janai, J., Güney, F., Behl, A., Geiger, A.: Computer vision for autonomous vehicles: problems, datasets and state-of-the-art. CoRR (2017). arXiv:1704.05519
[36]
Kuutti S et al. A survey of the state-of-the-art localization techniques and their potentials for autonomous vehicle applications IEEE Internet Things J. 2018 5 2 829-846
[37]
Borg M et al. Safely entering the deep: a review of verification and validation for machine learning and a challenge elicitation in the automotive industry J. Automot. Softw. Eng. 2019 1 1-19
[38]
Calikli, G., Bener, A.: Empirical analyses of the factors affecting confirmation bias and the effects of confirmation bias on software developer/tester performance. Paper presented at Proc. 6th Int. Conf. on Predictive Models in Software Engineering (PROMISE 2010) (2010)
[39]
Whang SE and Lee J-G Data collection and quality challenges for deep learning Proc VLDB Endow. 2020 13 12 3429-3432
[40]
Jo, J., Bengio, Y.: Measuring the Tendency of CNNs to Learn Surface Statistical Regularities. CoRR (2017). arXiv:1711.11561
[41]
Bengio, Y.: Priors for deep learning of semantic representations. Keynote at ACM/IEEE Int. Conf. on Model Driven Engineering Languages and Systems (MODELS) (2020)
[42]
Barredo Arrieta A et al. Explainable artificial intelligence (XAI): concepts, taxonomies, opportunities and challenges toward responsible AI Inf. Fusion 2020 58 82-115
[43]
Szegedy, C., et al.: Intriguing Properties of Neural Networks. CoRR (2013). Appeared in ICLR 2014. arXiv:1312.6199
[44]
Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and Harnessing Adversarial Examples. CoRR (2014). Appeared in ICLR 2015. arXiv:1412.6572
[45]
Nguyen, A., Yosinski, J., Clune, J.: Deep neural networks are easily fooled: high confidence predictions for unrecognizable images. Paper presented at IEEE Conf. on Computer Vision and Pattern Recognition (CVPR 2015) (2015)
[46]
Pei, K., Cao, Y., Yang, J., Jana, S.: DeepXplore: automated whitebox testing of deep learning systems. Paper presented at 26th Symposium on Operating Systems Principles (SOSP 2017) (2017)
[47]
Tian, Y., Pei, K., Jana, S., Ray, B.: DeepTest: automated testing of deep-neural-network-driven autonomous cars. Paper presented at 40th Int. Conf. on Software Engineering (NeurIPS 2018) (2018)
[48]
Ma, L., et al.: DeepGauge: multi-granularity testing criteria for deep learning systems. Paper presented at 33rd ACM/IEEE Int. Conf. on Automated Software Engineering (ASE 2018) (2018)
[49]
Zhang, M., Zhang, Y., Zhang, L., Liu, C., Khurshid, S.: DeepRoad: GAN-based metamorphic testing and input validation framework for autonomous driving systems. Paper presented at 33rd ACM/IEEE Int. Conf. on Automated Software Engineering (ASE 2018) (2018)
[50]
Sun, Y., et al.: DeepConcolic: testing and debugging deep neural networks. Paper presented at 41st Int. Conf. on Software Engineering (ICSE 2019) (2019)
[51]
Xie, X., et al.: DeepHunter: a coverage-guided fuzz testing framework for deep neural networks. Paper presented at 28th ACM SIGSOFT Int. Symposium on Software Testing and Analysis (2019)
[52]
Langford, M.A., Cheng, B. H.C.: Enhancing learning-enabled software systems to address environmental uncertainty. Paper presented at 16th IEEE Int. Conf. on Autonomic Computing (ICAC 2019) (2019)
[53]
Odena, A., Olsson, C., Andersen, D., Goodfellow, I.: TensorFuzz: debugging neural networks with coverage-guided fuzzing. Paper presented at 36th Int. Conf. on Machine Learning (PMLR 2019) (2019)
[54]
Berend, D., et al.: cats are not fish: deep learning testing calls for out-of-distribution awareness. Paper presented at 35th IEEE/ACM Int. Conf. on Automated Software Engineering (ASE 2020) (2020)
[55]
Ma W, Papadakis M, Tsakmalis A, Cordy M, and Traon YL Test selection for deep learning systems ACM Trans. Softw. Eng. Methodol. 2021 30 2 1-22
[56]
Naeem Irfan M, Oriat C, and Groz R Model inference and testing Adv. Comput. 2013 89 89-139
[57]
Fraser G and Walkinshaw N Assessing and generating test sets in terms of behavioural adequacy Softw. Test. Verif. Reliab. 2015 25 8 749-780
[58]
Papadopoulos, P., Walkinshaw, N.: Black-box test generation from inferred models. Paper presented at 4th Int. Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (RAISE 2015) (2015)
[59]
Aichernig BK et al. Gaston C, Kosmatov N, Le Gall P, et al. Learning a behavior model of hybrid systems through combining model-based testing and machine learning Testing Software and Systems 2019 Cham Springer
[60]
Gawlikowski, J., et al.: A survey of uncertainty in deep neural networks. CoRR (2021). arXiv:2107.03342
[61]
Cortés-Ciriano I and Bender A Deep confidence: a computationally efficient framework for calculating reliable prediction errors for deep neural networks J. Chem. Inf. Model. 2018 59 3 1269-1281
[62]
Brun Y et al. Engineering self-adaptive systems through feedback loops 2009 Berlin, Heidelberg Springer 48-70
[63]
IBM.: an architectural blueprint for autonomic computing. Tech. Rep. 3rd ed., IBM (2005). https://www-03.ibm.com/autonomic/pdfs/AC%20Blueprint%20White%20Paper%20V7.pdf
[64]
Arcaini, P., Riccobene, E., Scandurra, P.: Modeling and analyzing MAPE-K feedback loops for self-adaptation. Paper presented at 10th Int. Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2015) (2015)
[65]
Cheng, S.-W.: Rainbow: Cost-effective software architecture-based self-adaptation. Ph.D. thesis, Carnegie Mellon University (2008)
[66]
Walsh, W.E., Tesauro, G., Kephart, J.O., Das, R.: Utility functions in autonomic systems. Paper presented at Int. Conf. on Autonomic Computing (ICAC 2004) (2004)
[67]
deGrandis, P., Valetto, G.: Elicitation and utilization of application-level utility functions. Paper presented at 6th Int. Conf. on Autonomic Computing (ICAC 2009) (2009)
[68]
Object Management Group. Structured assurance case metamodel (SACM) Version 2.1. Tech. Rep., OMG (2020). https://www.omg.org/spec/SACM
[69]
Goodenough, J., Weinstock, C., Klein, A.: Toward a Theory of Assurance Case Confidence. Tech. Rep. CMU/SEI-2012-TR-002, Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA (2012). https://resources.sei.cmu.edu/library/asset-view.cfm?AssetID=28067
[70]
ACWG.: Goal structuring notation community standard (Version 2). Tech. Rep., Assurance Case Working Group, Safety-Critical Systems Club (2018). https://scsc.uk/r141B:1
[71]
Lapouchnian, A.: goal-oriented requirements engineering: an overview of the current research. Tech. Rep., University of Toronto (2005). http://www.cs.utoronto.ca/~alexei/pub/Lapouchnian-Depth.pdf
[72]
Letier, E., van Lamsweerde, A.: Reasoning about partial goal satisfaction for requirements and design engineering. Paper presented at 12th ACM SIGSOFT Int. Symp. on Foundations of Software Engineering (SIGSOFT 2004/FSE-12) (2004)
[73]
Whittle, J., Sawyer, P., Bencomo, N., Cheng, B.H.C., Bruel, J.-M.: RELAX: incorporating uncertainty into the specification of self-adaptive systems. Paper presented at 17th IEEE Int. Requirements Engineering Conf. (RE 2009) (2009)
[74]
Fredericks EM, DeVries B, and Cheng BHC Autorelax: automatically relaxing a goal model to address uncertainty Empir. Softw. Engg. 2014 19 5 1466-1501
[75]
Zadeh L Fuzzy logic Computer 1988 21 4 83-93
[76]
Hájek P Metamathematics of Fuzzy Logic 2013 Dordrecht Springer
[77]
Zadeh, L.A.: Fuzzy Sets (1996)
[78]
Clarke EM Jr, Grumberg O, Kroening D, Peled D, and Veith H Model Checking 2018 2 Cambridge MIT Press
[79]
Baresi, L., Pasquale, L., Spoletini, P.: Fuzzy goals for requirements-driven adaptation. Paper presented at 18th IEEE Int. Requirements Engineering Conf. (RE 2010) (2010)
[80]
Ferreira, R.S., Arlat, J., Guiochet, J., Waeselynck, H.: Benchmarking safety monitors for image classifiers with machine learning, 7–16 (IEEE, 2021)
[81]
Sokolova M and Lapalme G A systematic analysis of performance measures for classification tasks Inf. Process. Manag. 2009 45 427-437
[82]
Eiben AE and Smith JE Introduction to Evolutionary Computing 2015 2 Berlin, Heidelberg Springer-Verlag
[83]
Cheng, S.-W., Garlan, D., Schmerl, B.: Architecture-Based Self-Adaptation in the Presence of Multiple Objectives. Paper presented at Int. Workshop on Self-Adaptation and Self-Managing Systems (SEAMS (2006) (2006)
[84]
Palmerino, J., Yu, Q., Desell, T., Krutz, D.: Improving the decision-making process of self-adaptive systems by accounting for tactic volatility. Paper presented at 34th IEEE/ACM Int. Conf. on Automated Software Engineering (ASE 2019) (2019)
[85]
Zhang, J., Cheng, B.H.C.: Model-based development of dynamically adaptive software. Paper presented at 28th Int. Conf. on Software Engineering (ICSE 2006) (2006)
[86]
Kramer J and Magee J The evolving philosophers problem: dynamic change management IEEE Trans. Softw. Eng. 1990 16 11 1293-1306
[87]
Melenbrink N, Werfel J, and Menges A On-site autonomous construction robots: towards unsupervised building Autom. Constr. 2020 119
[88]
Malone, D.: Rovers set to invade construction jobsites (2019). https://www.bdcnetwork.com/rovers-set-invade-construction-jobsites
[89]
Weyns, D., Holvoet, T., Schelfthout, K., Wielemans, J.: Applying multi-agent systems in practice, decentralized control of automatic guided vehicles (2008)
[90]
Dersten, S., Wallin, P., Fröberg, J., Axelsson, J.: Analysis of the information needs of an autonomous hauler in a quarry site. Paper presented at 11th System of Systems Engineering Conf. (SoSE 2016) (2016)
[91]
Goldfain B et al. AutoRally: an open platform for aggressive autonomous driving IEEE Control Syst. Mag. 2019 39 1 26-55
[92]
Koenig, N., Howard, A.: Design and use paradigms for gazebo, an open-source multi-robot simulator. Paper presented at IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (2004)
[93]
Hartley RI and Sturm P Triangulation Comput. Vis. Image Underst. 1997 68 2 146-157
[94]
Rusu, R.B., Cousins, S.: 3D is here: point cloud library (PCL). Paper presented at IEEE Int. Conf. on Robotics and Automation (ICRA 2011) (2011)
[95]
Obstacle Avoidance and Robot Footprint Model (2019). http://wiki.ros.org/teb_local_planner/Tutorials/Obstacle%20Avoidance%20and%20Robot%20Footprint%20Model
[96]
Dalpiaz, F., Borgida, A., Horkoff, J., Mylopoulos, J.: Runtime goal models: keynote. Paper presented at 7th IEEE Int. Conf. on Research Challenges in Information Science (RCIS 2013) (2013)
[97]
Huang, W., et al.: Formal verification of robustness and resilience of learning-enabled state estimation systems for robotics. CoRR (2020). arXiv:2010.08311
[98]
Gu, R., Marinescu, R., Seceleanu, C., Lundqvist, K.: Formal verification of an autonomous wheel loader by model checking. Paper presented at 6th Conf. on Formal Methods in Software Engineering (FormaliSE 2018) (2018)
[99]
Ramos L et al. The RoCS framework to support the development of autonomous robots J. Softw. Eng. Res. Dev. 2019 7 1-14
[100]
Weyns, D., Iftikhar, M.U.: Model-based simulation at runtime for self-adaptive systems. Paper presented at 15th Int. Conf. on Autonomic Computing (ICAC 2016) (2016)
[101]
Calinescu R et al. Engineering trustworthy self-adaptive software with dynamic assurance cases IEEE Trans. Softw. Eng. 2018 44 11 1039-1069
[102]
Cheng, B.H.C., Clark, R.J., Fleck, J.E., Langford, M.A., McKinley, P.K.: AC-ROS: assurance case driven adaptation for the robot operating system (2020). Paper presented at 23rd Int. Conf. on Model Driven Engineering Languages and Systems (MODELS 2020)
[103]
Song Q, Shepperd M, Cartwright M, and Mair C Software defect association mining and defect correction effort prediction IEEE Trans. Softw. Eng. 2006 32 2 69-82
[104]
Rodriguez D, Ruiz R, Riquelme JC, and Harrison R A study of subgroup discovery approaches for defect prediction Inf. Softw. Technol. 2013 55 10 1810-1822
[105]
Smith, C., Denney, E., Pai, G.: Hazard contribution modes of machine learning components. Tech. Rep., OSTI (2020). https://www.osti.gov/biblio/1606667. (AAAI Workshop: SafeAI 2020)
[106]
Ramirez, A.J., Cheng, B.H.C.: Automatic derivation of utility functions for monitoring software requirements. Paper presented at 14th Int. Conf. on Model Driven Engineering Languages and Systems (MODELS 2011) (2011)
[107]
Bencomo, N., Belaggoun, A.: Supporting decision-making for self-adaptive systems: from goal models to dynamic decision networks. Paper presented at Int. Working Conf. on Requirements Engineering Foundation for Software Quality (REFSQ) (2013) (2013)
[108]
Letier, E., Stefan, D., Barr, E.T.: Uncertainty, risk, and information value in software requirements and architecture. Paper presented at 36th Int. Conf. on Software Engineering (ICSE 2014) (2014)
[109]
Asaadi, E., Denney, E., Pai, G.: quantifying assurance in learning-enabled systems. Paper presented at Int. Conf. on Computer Safety, Reliability, and Security (SAFECOMP 2020) (2020)
[110]
Boursinos, D., Koutsoukos, X.: Assurance Monitoring of Learning-Enabled Cyber-Physical Systems Using Inductive Conformal Prediction Based on Distance Learning. Artif. Intell. Eng. Des. Anal. Manuf. 35(2), 251–264 (2021).
[111]
Weyns, D., et al.: towards better adaptive systems by combining MAPE, control theory, and machine learning. Paper presented at 16th Int. Symp. on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2021) (2021)
[112]
Machin M et al. SMOF: a safety monitoring framework for autonomous systems IEEE Trans. Syst. Man Cybern. Syst. 2016 48 5 702-715
[113]
Jahan, S., et al.: MAPE-K/MAPE-SAC: An Interaction Framework for Adaptive Systems with Security Assurance Cases. Futur. Gener. Comput. Syst. 109, 197–209 (2020).
Recommendations
Model-based design for CPS with learning-enabled components
DESTION '19: Proceedings of the Workshop on Design Automation for CPS and IoTRecent advances in machine learning led to the appearance of Learning-Enabled Components (LECs) in Cyber-Physical Systems. LECs are being evaluated and used for various, complex functions including perception and control. However, very little tool ...
CPS Design with Learning-Enabled Components: A Case Study
RSP '19: Proceedings of the 30th International Workshop on Rapid System Prototyping (RSP'19)Cyber-Physical Systems (CPS) are used in many applications where they must perform complex tasks with a high degree of autonomy in uncertain environments. Traditional design flows based on domain knowledge and analytical models are often impractical for ...
Exploring the Impact of Artificial Intelligence Learning Platforms on Interest in and Attitudes Toward Learning
Innovative Technologies and LearningAbstractArtificial Intelligence (AI) learning technologies and research in deep learning are rapidly expanding, and there is a lack of AI learning platforms to support students in related courses. At present, an understanding of the impact of learning on ...
Comments
Information & Contributors
Information
Published In
© The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature 2023. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 18 March 2023
Accepted: 30 January 2023
Revision received: 26 January 2023
Received: 14 March 2022
Author Tags
Qualifiers
- Research-article
Funding Sources
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 04 Oct 2024
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in