Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3368089.3409737acmconferencesArticle/Chapter ViewAbstractPublication PagesfseConference Proceedingsconference-collections
research-article

Mining assumptions for software components using machine learning

Published: 08 November 2020 Publication History

Abstract

Software verification approaches aim to check a software component under analysis for all possible environments. In reality, however, components are expected to operate within a larger system and are required to satisfy their requirements only when their inputs are constrained by environment assumptions. In this paper, we propose EPIcuRus, an approach to automatically synthesize environment assumptions for a component under analysis (i.e., conditions on the component inputs under which the component is guaranteed to satisfy its requirements). EPIcuRus combines search-based testing, machine learning and model checking. The core of EPIcuRus is a decision tree algorithm that infers environment assumptions from a set of test results including test cases and their verdicts. The test cases are generated using search-based testing, and the assumptions inferred by decision trees are validated through model checking. In order to improve the efficiency and effectiveness of the assumption generation process, we propose a novel test case generation technique, namely Important Features Boundary Test (IFBT), that guides the test generation based on the feedback produced by machine learning. We evaluated EPIcuRus by assessing its effectiveness in computing assumptions on a set of study subjects that include 18 requirements of four industrial models. We show that, for each of the 18 requirements, EPIcuRus was able to compute an assumption to ensure the satisfaction of that requirement, and further, ≈78% of these assumptions were computed in one hour.

Supplementary Material

Auxiliary Teaser Video (fse20main-p485-p-teaser.mp4)
This material contains the Main Video (mp4, 1920×1080)
Auxiliary Presentation Video (fse20main-p485-p-video.mp4)
This material contains the Main Video (mp4, 1920×1080)

References

[1]
2019. Autopilot online documents. https://nl.mathworks.com/matlabcentral/ ifleexchange/41490-autopilot-demo-for-arp4754a-do-178c-and-do331?focused=6796756&tab=model. Accessed: 2019-08-07.
[2]
2019. DHC-2model. http://www.dutchroll.com. Accessed: 2019-10-24.
[3]
2020. Additional Material. https://github.com/SNTSVV/EPIcuRus
[4]
2020. fitctree. https://nl.mathworks.com/help/stats/fitctree.html
[5]
2020. QVtrace. https://qracorp.com/qvtrace/
[6]
2020. Simulink. https://nl.mathworks.com/products/simulink.html
[7]
United States. Federal Aviation Administration. 2009. Advanced Avionics Handbook. Aviation Supplies & Academics, Incorporated. https://books.google.lu/ books?id=2xGuPwAACAAJ
[8]
Rajeev Alur, Salar Moarref, and Ufuk Topcu. 2013. Counter-strategy guided refinement of GR(1) temporal logic specifications. In Formal Methods in ComputerAided Design, FMCAD. IEEE, 26-33.
[9]
Étienne André. 2019. What's decidable about parametric timed automata ? International Journal on Software Tools for Technology Transfer 21, 2 ( 2019 ), 203-219.
[10]
Yashwanth Annapureddy, Che Liu, Georgios Fainekos, and Sriram Sankaranarayanan. 2011. S-Taliro: A Tool for Temporal Logic Falsification for Hybrid Systems. In Proceedings of the 17th International Conference on Tools and Algorithms for the Construction and Analysis of Systems: Part of the Joint European Conferences on Theory and Practice of Software (TACAS'11/ETAPS'11). SpringerVerlag, Berlin, Heidelberg, 254-257.
[11]
Yashwanth Annpureddy, Che Liu, Georgios Fainekos, and Sriram Sankaranarayanan. 2011. S-taliro: A tool for temporal logic falsification for hybrid systems. In International Conference on Tools and Algorithms for the Construction and Analysis of Systems. Springer, 254-257.
[12]
Andrea Arcuri and Lionel C. Briand. 2011. Adaptive random testing: an illusion of efectiveness?. In International Symposium on Software Testing and Analysis, ISSTA. ACM, 265-275.
[13]
Andrea Arcuri, Muhammad Zohaib Z. Iqbal, and Lionel C. Briand. 2012. Random Testing: Theoretical Results and Practical Implications. IEEE Trans. Software Eng. 38, 2 ( 2012 ), 258-277. https://doi.org/10.1109/TSE. 2011.121
[14]
Aitor Arrieta, Joseba Andoni Agirre, and Goiuria Sagardui. 2020. Seeding Strategies for Multi-Objective Test Case Selection: An Application on Simulation-Based Testing. In Genetic and Evolutionary Computation Conference. ACM.
[15]
Aitor Arrieta, Joseba Andoni Agirre, and Goiuria Sagardui. 2020. A Tool for the Automatic Generation of Test Cases and Oracles for Simulation Models Based on Functional Requirements. In International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE.
[16]
Aitor Arrieta, Shuai Wang, Urtzi Markiegi, Ainhoa Arruabarrena, Leire Etxeberria, and Goiuria Sagardui. 2019. Pareto eficient multi-objective black-box test case selection for simulation-based testing. Information and Software Technology 114 ( 2019 ), 137-154.
[17]
Wolfgang Banzhaf, Peter Nordin, Robert E Keller, and Frank D Francone. 1998. Genetic programming. Springer.
[18]
Howard Barringer and Dimitra Giannakopoulou. 2003. Proof Rules for Automated Compositional Verification through Learning. In In Proc. SAVCBS Workshop. 14-21.
[19]
Matthias Bernaerts, Bentley Oakes, Ken Vanherpen, Bjorn Aelvoet, Hans Vangheluwe, and Joachim Denil. 2019. Validating Industrial Requirements with a Contract-Based Approach. In International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C). IEEE, 18-27.
[20]
Marcello M. Bersani, Matteo Rossi, and Pierluigi San Pietro. 2016. A tool for deciding the satisfiability of continuous-time metric temporal logic. Acta Inf. 53, 2 ( 2016 ), 171-206. https://doi.org/10.1007/s00236-015-0229-y
[21]
R. V. Borges, A. d'Avila Garcez, L. C. Lamb, and B. Nuseibeh. 2011. Learning to adapt requirements specifications of evolving systems: (NIER track). In International Conference on Software Engineering (ICSE). IEEE.
[22]
Leo Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone. 1984. Classification and Regression Trees. Wadsworth.
[23]
Christos G. Cassandras and Stéphane Lafortune. 2008. Introduction to Discrete Event Systems, Second Edition. Springer. https://doi.org/10.1007/978-0-387-68612-7
[24]
Devendra K Chaturvedi. 2017. Modeling and simulation of systems using MATLAB and Simulink. CRC press.
[25]
Tsong Yueh Chen, Fei-Ching Kuo, Robert G. Merkel, and T. H. Tse. 2010. Adaptive Random Testing: The ART of test case diversity. J. Syst. Softw. 83, 1 ( 2010 ), 60-66.
[26]
Tsong Yueh Chen, Hing Leung, and I. K. Mak. 2004. Adaptive Random Testing. In Advances in Computer Science-ASIAN. Springer, 320-329.
[27]
Jamieson M. Cobleigh, Dimitra Giannakopoulou, and Corina S. Pasareanu. 2003. Learning Assumptions for Compositional Verification. In Tools and Algorithms for the Construction and Analysis of Systems TACAS. Springer, 331-346.
[28]
Luca de Alfaro and Thomas A. Henzinger. 2001. Interface Automata. SIGSOFT Softw. Eng. Notes 26, 5 (Sept. 2001 ), 12. https://doi.org/10.1145/503271.503226
[29]
Luca de Alfaro and Thomas A. Henzinger. 2001. Interface Theories for Component-Based Design. In Embedded Software. Springer, 148-165.
[30]
Leonardo De Moura and Nikolaj Bjørner. 2008. Z3: An eficient SMT solver. In International conference on Tools and Algorithms for the Construction and Analysis of Systems. Springer, 337-340.
[31]
Patricia Derler, Edward A. Lee, Stavros Tripakis, and Martin Törngren. 2013. Cyber-Physical System Design Contracts. In International Conference on CyberPhysical Systems. ACM.
[32]
Patricia Derler, Edward A Lee, and Alberto Sangiovanni Vincentelli. 2012. Modeling Cyber-Physical Systems. Proc. IEEE 100, 1 ( 2012 ), 13-28. https: //doi.org/10.1109/JPROC. 2011.2160929
[33]
Gidon Ernst, Paolo Arcaini, Alexandre Donze, Georgios Fainekos, Logan Mathesen, Giulia Pedrielli, Shakiba Yaghoubi, Yoriyuki Yamagata, and Zhenya Zhang. 2019. ARCH-COMP 2019 Category Report: Falsification. EPiC Series in Computing 61 ( 2019 ), 129-140.
[34]
Goran Frehse, Colas Le Guernic, Alexandre Donzé, Scott Cotton, Rajarshi Ray, Olivier Lebeltel, Rodolfo Ripado, Antoine Girard, Thao Dang, and Oded Maler. 2011. SpaceEx: Scalable verification of hybrid systems. In International Conference on Computer Aided Verification. Springer, 379-395.
[35]
Khouloud Gaaloul, Claudio Menghi, Shiva Nejati, and Lionel Briand. 2020. EPIcuRus. Zenodo. https://doi.org/10.5281/zenodo.3872902
[36]
Dimitra Giannakopoulou, Corina S. Pasareanu, and Howard Barringer. 2002. Assumption Generation for Software Component Verification. In International Conference on Automated Software Engineering (ASE). IEEE.
[37]
Dimitra Giannakopoulou, Corina S. Pasareanu, and Colin Blundell. 2008. Assumeguarantee testing for software components. IET Software 2, 6 ( 2008 ), 547-562. https://doi.org/10.1049/iet-sen: 20080012
[38]
Dimitra Giannakopoulou, Corina S Pasareanu, and Jamieson M Cobleigh. 2004. Assume-guarantee verification of source code with design-level assumptions. In International Conference on Software Engineering. IEEE, 211-220.
[39]
Mark Harman, S. Afshin Mansouri, and Yuanyuan Zhang. 2012. Search-based software engineering: Trends, techniques and applications. ACM Comput. Surv. 45, 1 ( 2012 ), 11 : 1-11 : 61. https://doi.org/10.1145/2379776.2379787
[40]
Thomas A. Henzinger, Shaz Qadeer, and Sriram K. Rajamani. 1998. You assume, we guarantee: Methodology and case studies. In Computer Aided Verification. Springer, 440-451.
[41]
Rubing Huang, Weifeng Sun, Yinyin Xu, Haibo Chen, Dave Towey, and Xin Xia. 2019. A survey on adaptive random testing. IEEE Transactions on Software Engineering ( 2019 ).
[42]
Xiaoqing Jin, Alexandre Donzé, Jyotirmoy V. Deshmukh, and Sanjit A. Seshia. 2013. Mining requirements from closed-loop control models. In international conference on Hybrid systems: computation and control, HSCC. ACM.
[43]
Xiaoqing Jin, Alexandre Donzé, Jyotirmoy V. Deshmukh, and Sanjit A. Seshia. 2015. Mining Requirements From Closed-Loop Control Models. IEEE Trans. on CAD of Integrated Circuits and Systems 34, 11 ( 2015 ), 1704-1717. https: //doi.org/10.1109/TCAD. 2015.2421907
[44]
James Kapinski, Jyotirmoy V Deshmukh, Xiaoqing Jin, Hisahiro Ito, and Ken Butts. 2016. Simulation-Based Approaches for Verification of Embedded Control Systems: An Overview of Traditional and Advanced Modeling, Testing, and Verification Techniques. IEEE Control Systems Magazine 36, 6 ( 2016 ), 45-64.
[45]
Wenchao Li, Lili Dworkin, and Sanjit A. Seshia. 2011. Mining assumptions for synthesis. In International Conference on Formal Methods and Models. IEEE, 43-50.
[46]
Shahar Maoz, Jan Oliver Ringert, and Rafi Shalom. 2019. Symbolic repairs for GR (1) specifications. In International Conference on Software Engineering (ICSE). IEEE, 1016-1026.
[47]
Reza Matinnejad, Shiva Nejati, and Lionel C Briand. 2017. Automated testing of hybrid Simulink/Stateflow controllers: industrial case studies. In Foundations of Software Engineering. ACM, 938-943.
[48]
Anastasia Mavridou, Hamza Bourbouh, Pierre-Loïc Garoche, Dimitra Giannakopoulou, Tom Pressburger, and Johann Schumann. 2020. Bridging the Gap Between Requirements and Simulink Model Analysis. In Requirements Engineering: Foundation for Software Quality (REFSQ), Companion Proceedings. Springer.
[49]
Claudio Menghi, Marcello M. Bersani, Matteo Rossi, and Pierluigi San Pietro. 2020. Model Checking MITL formulae on Timed Automata: a Logic-Based Approach. Transactions on Computational Logic ( 2020 ).
[50]
Claudio Menghi, Shiva Nejati, Lionel C. Briand, and Yago Isasi Parache. 2020. Approximation-Refinement Testing of Compute-Intensive Cyber-Physical Models: An Approach Based on System Identification. In International Conference on Software Engineering (ICSE). IEEE / ACM.
[51]
Claudio Menghi, Shiva Nejati, Khouloud Gaaloul, and Lionel C. Briand. 2019. Generating automated and online test oracles for Simulink models with continuous and uncertain behaviors. In European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE. ACM.
[52]
Claudio Menghi, Paola Spoletini, Marsha Chechik, and Carlo Ghezzi. 2018. Supporting Verification-Driven Incremental Distributed Design of Components. In Fundamental Approaches to Software Engineering FASE. Springer, 169-188.
[53]
Claudio Menghi, Paola Spoletini, Marsha Chechik, and Carlo Ghezzi. 2019. A verification-driven framework for iterative design of controllers. Formal Asp. Comput. 31, 5 ( 2019 ), 459-502. https://doi.org/10.1007/s00165-019-00484-1
[54]
Christoph Molnar. 2019. Interpretable Machine Learning. https://christophm. github.io/interpretable-ml-book/.
[55]
Christoph Molnar. 2019. Interpretable machine learning. Lulu. com.
[56]
Shiva Nejati, Khouloud Gaaloul, Claudio Menghi, Lionel C Briand, Stephen Foster, and David Wolfe. 2019. Evaluating model testing and model checking for finding requirements violations in Simulink models. In European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE. ACM, 1015-1025.
[57]
Pierluigi Nuzzo, John B. Finn, Antonio Iannopollo, and Alberto L. SangiovanniVincentelli. 2014. Contract-based design of control protocols for safety-critical cyber-physical systems. In Design, Automation & Test in Europe Conference & Exhibition, (DATE). European Design and Automation Association.
[58]
Pierluigi Nuzzo, Huan Xu, Necmiye Ozay, John B. Finn, Alberto L. SangiovanniVincentelli, Richard M. Murray, Alexandre Donzé, and Sanjit A. Seshia. 2014. A Contract-Based Methodology for Aircraft Electric Power System Design. IEEE Access 2 ( 2014 ), 1-25. https://doi.org/10.1109/ACCESS. 2013.2295764
[59]
Peter J Ramadge and W Murray Wonham. 1987. Supervisory control of a class of discrete event processes. SIAM journal on control and optimization 25, 1 ( 1987 ), 206-230.
[60]
Peter JG Ramadge and W Murray Wonham. 1989. The control of discrete event systems. Proc. IEEE 77, 1 ( 1989 ), 81-98.
[61]
Alberto Sangiovanni-Vincentelli, Werner Damm, and Roberto Passerone. 2012. Taming Dr. Frankenstein: Contract-based design for cyber-physical systems. European journal of control 18, 3 ( 2012 ), 217-238.
[62]
Armando Solar-Lezama, Liviu Tancau, Rastislav Bodík, Sanjit A. Seshia, and Vijay A. Saraswat. 2006. Combinatorial sketching for finite programs. In International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS). ACM.
[63]
X. Sun, P. Nuzzo, C. Wu, and A. Sangiovanni-Vincentelli. 2009. Contract-based system-level composition of analog circuits. In Design Automation Conference. ACM.
[64]
Ashish Tiwari. 2012. HybridSAL Relational Abstracter. In Computer Aided Verification (CAV). Springer.
[65]
Cumhur Erkan Tuncali, Georgios Fainekos, Danil Prokhorov, Hisahiro Ito, and James Kapinski. 2019. Requirements-driven test generation for autonomous vehicles with machine learning components. IEEE Transactions on Intelligent Vehicles 5, 2 ( 2019 ), 265-280.
[66]
S. Varrette, P. Bouvry, H. Cartiaux, and F. Georgatos. 2014. Management of an Academic HPC Cluster: The UL Experience. In Proc. of the 2014 Intl. Conf. on High Performance Computing & Simulation (HPCS 2014 ). IEEE, Bologna, Italy, 959-967.
[67]
Ian H. Witten, Eibe Frank, Mark A. Hall, and Christopher J. Pal. 2016. Data Mining, Fourth Edition: Practical Machine Learning Tools and Techniques (4th ed.). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.

Cited By

View all
  • (2024)Enhancing Automata Learning with Statistical Machine Learning: A Network Security Case StudyProceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems10.1145/3640310.3674087(172-182)Online publication date: 22-Sep-2024
  • (2024)Insights into System Failures: ML-Assisted Testing and Failure Models for Cyber-Physical Systems2024 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST60714.2024.00055(460-462)Online publication date: 27-May-2024
  • (2023)Test Generation Strategies for Building Failure Models and Explaining Spurious FailuresACM Transactions on Software Engineering and Methodology10.1145/363824633:4(1-32)Online publication date: 21-Dec-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ESEC/FSE 2020: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering
November 2020
1703 pages
ISBN:9781450370431
DOI:10.1145/3368089
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 November 2020

Permissions

Request permissions for this article.

Check for updates

Badges

Author Tags

  1. Decision trees
  2. Environment assumptions
  3. Machine learning
  4. Model checking
  5. Search-based software testing

Qualifiers

  • Research-article

Funding Sources

Conference

ESEC/FSE '20
Sponsor:

Acceptance Rates

Overall Acceptance Rate 112 of 543 submissions, 21%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)42
  • Downloads (Last 6 weeks)3
Reflects downloads up to 09 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Enhancing Automata Learning with Statistical Machine Learning: A Network Security Case StudyProceedings of the ACM/IEEE 27th International Conference on Model Driven Engineering Languages and Systems10.1145/3640310.3674087(172-182)Online publication date: 22-Sep-2024
  • (2024)Insights into System Failures: ML-Assisted Testing and Failure Models for Cyber-Physical Systems2024 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST60714.2024.00055(460-462)Online publication date: 27-May-2024
  • (2023)Test Generation Strategies for Building Failure Models and Explaining Spurious FailuresACM Transactions on Software Engineering and Methodology10.1145/363824633:4(1-32)Online publication date: 21-Dec-2023
  • (2023)Search-Based Software Testing Driven by Automatically Generated and Manually Defined Fitness FunctionsACM Transactions on Software Engineering and Methodology10.1145/362474533:2(1-37)Online publication date: 23-Dec-2023
  • (2023)Learning Non-robustness using Simulation-based Testing: a Network Traffic-shaping Case Study2023 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST57152.2023.00043(386-397)Online publication date: Apr-2023
  • (2023)Contract-Based Specification Refinement and Repair for Mission Planning2023 IEEE/ACM 11th International Conference on Formal Methods in Software Engineering (FormaliSE)10.1109/FormaliSE58978.2023.00011(29-38)Online publication date: May-2023
  • (2023)Computing Adequately Permissive Assumptions for SynthesisTools and Algorithms for the Construction and Analysis of Systems10.1007/978-3-031-30820-8_15(211-228)Online publication date: 22-Apr-2023
  • (2022)Consistent Scene Graph Generation by Constraint OptimizationProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering10.1145/3551349.3560433(1-13)Online publication date: 10-Oct-2022
  • (2022)Combining Genetic Programming and Model Checking to Generate Environment AssumptionsIEEE Transactions on Software Engineering10.1109/TSE.2021.310181848:9(3664-3685)Online publication date: 1-Sep-2022
  • (2022)Machine learning‐based test oracles for performance testing of cyber‐physical systems: An industrial case study on elevators dispatching algorithmsJournal of Software: Evolution and Process10.1002/smr.246534:11Online publication date: 25-May-2022
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media