Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3377811.3380381acmconferencesArticle/Chapter ViewAbstractPublication PagesicseConference Proceedingsconference-collections
research-article
Public Access

SLEMI: equivalence modulo input (EMI) based mutation of CPS models for finding compiler bugs in Simulink

Published: 01 October 2020 Publication History

Abstract

Finding bugs in commercial cyber-physical system development tools (or "model-based design" tools) such as MathWorks's Simulink is important in practice, as these tools are widely used to generate embedded code that gets deployed in safety-critical applications such as cars and planes. Equivalence Modulo Input (EMI) based mutation is a new twist on differential testing that promises lower use of computational resources and has already been successful at finding bugs in compilers for procedural languages. To provide EMI-based mutation for differential testing of cyber-physical system (CPS) development tools, this paper develops several novel mutation techniques. These techniques deal with CPS language features that are not found in procedural languages, such as an explicit notion of execution time and zombie code, which combines properties of live and dead procedural code. In our experiments the most closely related work (SLforge) found two bugs in the Simulink tool. In comparison, SLEMI found a super-set of issues, including 9 confirmed as bugs by MathWorks Support.

References

[1]
Alessandro Abate, Iury Bessa, Dario Cattaruzza, Lennon C. Chaves, Lucas C. Cordeiro, Cristina David, Pascal Kesseli, Daniel Kroening, and Elizabeth Polgreen. 2017. DSSynth: An automated digital controller synthesis tool for physical plants. In Proc. 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). 919--924.
[2]
Rajeev Alur. 2011. Formal verification of hybrid systems. In Proc. 11th International Conference on Embedded Software, (EMSOFT) 2011. ACM, 273--278.
[3]
Rajeev Alur, Aditya Kanade, S. Ramesh, and K. C. Shashidhar. 2008. Symbolic analysis for improving simulation coverage of Simulink/Stateflow models. In Proc. 8th ACM & IEEE International Conference on Embedded Software (EMSOFT). ACM, 89--98.
[4]
Stanley Bak, Sergiy Bogomolov, and Taylor T. Johnson. 2015. HYST: A Source Transformation and Translation Tool for Hybrid Automaton Models. In Proc. 18th International Conference on Hybrid Systems: Computation and Control (HSCC). ACM, 128--133.
[5]
Olivier Bouissou and Alexandre Chapoutot. 2012. An Operational Semantics for Simulink's Simulation Engine. In Proc. 13th ACM SIGPLAN/SIGBED International Conference on Languages, Compilers, Tools and Theory for Embedded Systems (LCTES). ACM, 129--138.
[6]
Angelo Brillout, Nannan He, Michele Mazzucchi, Daniel Kroening, Mitra Purandare, Philipp Rümmer, and Georg Weissenbacher. 2010. Mutation-Based Test Case Generation for Simulink Models. In Formal Methods for Components and Objects: 8th International Symposium, FMCO 2009, November 4--6, 2009. Revised Selected Papers. Springer, 208--227.
[7]
Lennon C. Chaves, Iury Bessa, Lucas C. Cordeiro, Daniel Kroening, and Eddie Batista de Lima Filho. 2017. Verifying digital systems with MATLAB. In Proc. 26th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). 388--391.
[8]
Junjie Chen, Wenxiang Hu, Dan Hao, Yingfei Xiong, Hongyu Zhang, Lu Zhang, and Bing Xie. 2016. An Empirical Comparison of Compiler Testing Techniques. In Proc. 38th International Conference on Software Engineering (ICSE). ACM, 180--190.
[9]
Shafiul Azam Chowdhury. 2019. Automated Testing of a Commercial Cyber-Physical System Development Tool Chain. Ph.D. Dissertation. University of Texas at Arlington.
[10]
Shafiul Azam Chowdhury et al. 2018. SLforge web site. https://github.com/verivital/slsf_randgen/wiki. Accessed Jan 2020.
[11]
Shafiul Azam Chowdhury et al. 2020. SLEMI web site. https://github.com/shafiul/slemi. Accessed Jan 2020.
[12]
Shafiul Azam Chowdhury, Taylor T. Johnson, and Christoph Csallner. 2016. CyFuzz: A differential testing framework for cyber-physical systems development environments. In Proc. 6th Workshop on Design, Modeling and Evaluation of Cyber Physical Systems (CyPhy). Springer.
[13]
Shafiul Azam Chowdhury, Soumik Mohian, Sidharth Mehra, Siddhant Gawsane, Taylor T. Johnson, and Christoph Csallner. 2018. Automatically finding bugs in a commercial cyber-physical system development tool chain with SLforge. In Proc. 40th ACM/IEEE International Conference on Software Engineering (ICSE). ACM, 981--992.
[14]
Shafiul Azam Chowdhury, Lina Sera Varghese, Soumik Mohian, Taylor T. Johnson, and Christoph Csallner. 2018. A curated corpus of Simulink models for model-based empirical studies. In Proc. 4th International Workshop on Software Engineering for Smart Cyber-Physical Systems (SEsCPS). ACM, 45--48.
[15]
Mirko Conrad. 2009. Testing-based translation validation of generated code in the context of IEC 61508. Formal Methods in System Design 35, 3 (Dec. 2009), 389--401.
[16]
Joachim Denil, Pieter J. Mosterman, and Hans Vangheluwe. 2014. Rule-based model transformation for, and in Simulink. In Proc. Symposium on Theory of Modeling and Simulation (TMS). ACM, 314--321.
[17]
Kyle Dewey, Jared Roesch, and Ben Hardekopf. 2015. Fuzzing the Rust Type-checker Using CLP (T). In Proc. 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 482--493.
[18]
Vijay D'Silva, Daniel Kroening, and Georg Weissenbacher. 2008. A survey of automated techniques for formal software verification. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 27, 7 (July 2008), 1165--1178.
[19]
dSPACE Inc. 2020. TargetLink. https://www.dspace.com/en/inc/home/products/sw/pcgs/targetli.cfm. Accessed Jan 2020.
[20]
Péter Fehér, Tamás Mészáros, László Lengyel, and Pieter J. Mosterman. 2013. Data type propagation in Simulink models with graph transformation. In Proc. 3rd Eastern European Regional Conference on the Engineering of Computer Based Systems (ECBS-EERC). IEEE, 127--137.
[21]
Kamran Ghani, John Clark, and Yuan Zhan. 2009. Comparing algorithms for search-based test data generation of MATLAB Simulink models. In Proc. IEEE Congress on Evolutionary Computation. 2940--2947.
[22]
Antoine Girard, A. Agung Julius, and George J. Pappas. 2008. Approximate Simulation Relations for Hybrid Systems. Discrete Event Dynamic Systems 18, 2 (2008), 163--179.
[23]
Christoph Guger, Alois Schlogl, Christa Neuper, Dirk Walterspacher, Thomas Strein, and Gert Pfurtscheller. 2001. Rapid prototyping of an EEG-based brain-computer interface (BCI). IEEE Transactions on Neural Systems and Rehabilitation Engineering 9, 1 (March 2001), 49--58.
[24]
Grégoire Hamon and John Rushby. 2007. An operational semantics for Stateflow. International Journal on Software Tools for Technology Transfer 9, 5 (2007), 447--456.
[25]
Nannan He, Philipp Rümmer, and Daniel Kroening. 2011. Test-case generation for embedded Simulink via formal concept analysis. In Proc. 48th Design Automation Conference (DAC). ACM, 224--229.
[26]
Wesley M. Johnston, J. R. Paul Hanna, and Richard J. Millar. 2004. Advances in dataflow programming languages. Comput. Surveys 36, 1 (March 2004), 1--34.
[27]
Aditya Kanade, Rajeev Alur, Franjo Ivancic, S. Ramesh, Sriram Sankaranarayanan, and K. C. Shashidhar. 2009. Generating and Analyzing Symbolic Traces of Simulink/Stateflow Models. In Proc. 21st International Conference on Computer Aided Verification (CAV). Springer, 430--445.
[28]
Vu Le, Mehrdad Afshari, and Zhendong Su. 2014. Compiler validation via equivalence modulo inputs. In Proc. ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). ACM, 216--226.
[29]
Vu Le, Chengnian Sun, and Zhendong Su. 2015. Finding Deep Compiler Bugs via Guided Stochastic Program Mutation. In Proc. 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA). ACM, 386--399.
[30]
Christopher Lidbury, Andrei Lascu, Nathan Chong, and Alastair F. Donaldson. 2015. Many-core Compiler Fuzzing. In Proc. 36th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). ACM, 65--76.
[31]
Bing Liu, Lucia, Shiva Nejati, and Lionel C. Briand. 2017. Improving fault localization for Simulink models using search-based testing and prediction models. In Proc. 24th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE.
[32]
MathWorks Inc. 2018. Products and Services. http://www.mathworks.com/products/. Accessed Jan 2020.
[33]
MathWorks Inc. 2020. About Data Types in Simulink. https://www.mathworks.com/help/simulink/ug/working-with-data-types.html. Accessed Jan 2020.
[34]
MathWorks Inc. 2020. Algebraic Loop Concepts --- MATLAB & Simulink. https://www.mathworks.com/help/simulink/ug/algebraic-loops.html. Accessed Jan 2020.
[35]
MathWorks Inc. 2020. Blocks for Which Sample Time is Not Recommended --- MATLAB & Simulink. https://www.mathworks.com/help/simulink/ug/sampletimehiding.html. Accessed Jan 2020.
[36]
MathWorks Inc. 2020. Choose a Solver --- MATLAB & Simulink. https://www.mathworks.com/help/simulink/ug/types-of-solvers.html. Accessed Jan 2020.
[37]
MathWorks Inc. 2020. Sample Time --- MATLAB & Simulink. https://www.mathworks.com/help/simulink/sample-time.html. Accessed Jan 2020.
[38]
MathWorks Inc. 2020. Simulink Coverage --- MATLAB & Simulink. https://www.mathworks.com/help/slcoverage/. Accessed Jan 2020.
[39]
MathWorks Inc. 2020. Simulink Design Verifier --- MATLAB & Simulink. https://www.mathworks.com/products/sldesignverifier.html. Accessed Jan 2020.
[40]
MathWorks Inc. 2020. Simulink Documentation --- Conditional Subsystem Initial Output Values. https://www.mathworks.com/help/simulink/ug/specify-or-inherit-conditional-subsystem-initial-values.html. Accessed Jan 2020.
[41]
MathWorks Inc. 2020. Simulink Documentation --- MATLAB & Simulink. http://www.mathworks.com/help/simulink/. Accessed Jan 2020.
[42]
Reza Matinnejad, Shiva Nejati, Lionel C. Briand, and Thomas Bruckmann. 2016. SimCoTest: A test suite generation tool for Simulink/Stateflow controllers. In Proc. 38th International Conference on Software Engineering, (ICSE). ACM, 585--588.
[43]
William M. McKeeman. 1998. Differential Testing for Software. Digital Technical Journal 10, 1 (1998), 100--107. http://www.hpl.hp.com/hpjournal/dtj/vol10num1/vol10num1art9.pdf
[44]
Pieter J. Mosterman, Justyna Zander, Grégoire Hamon, and Ben Denckla. 2009. Towards computational hybrid system semantics for time-based block diagrams. In Proc. 3rd IFAC Conference on Analysis and Design of Hybrid Systems (ADHS). Elsevier, 376--385.
[45]
Luan Viet Nguyen, Christian Schilling, Sergiy Bogomolov, and Taylor T. Johnson. 2015. Runtime Verification of Model-based Development Environments. In Proc. 15th International Conference on Runtime Verification (RV).
[46]
Bentley James Oakes. 2014. Optimizing Simulink Models. Technical Report SOCS-TR-2014.5. McGill University.
[47]
Marta Olszewska, Yanja Dajsuren, Harald Altinger, Alexander Serebrenik, Marina A. Waldén, and Mark G. J. van den Brand. 2016. Tailoring complexity metrics for Simulink models. In Proc. 10th European Conference on Software Architecture Workshops.
[48]
Vera Pantelic, Steven Postma, Mark Lawford, Monika Jaskolka, Bennett Mackenzie, Alexandre Korobkine, Marc Bender, Jeff Ong, Gordon Marks, and Alan Wassyng. 2017. Software engineering practices and Simulink: Bridging the gap. International Journal on Software Tools for Technology Transfer (STTT) 20, 1 (Feb. 2017), 95--117.
[49]
Akshay Rajhans, Srinath Avadhanula, Alongkrit Chutinan, Pieter J. Mosterman, and Fu Zhang. 2018. Graphical modeling of hybrid dynamics with Simulink and Stateflow. In Proc. 21st International Conference on Hybrid Systems: Computation and Control (HSCC). ACM, 247--252.
[50]
Pritam Roy, Paulo Tabuada, and Rupak Majumdar. 2011. Pessoa 2.0: A controller synthesis tool for cyber-physical systems. In Proc. 14th ACM International Conference on Hybrid Systems: Computation and Control (HSCC). 315--316.
[51]
Prahladavaradan Sampath, A. C. Rajeev, S. Ramesh, and K. C. Shashidhar. 2007. Testing Model-Processing Tools for Embedded Systems. In Proc. 13th IEEE RealTime and Embedded Technology and Applications Symposium. IEEE, 203--214.
[52]
Flash Sheridan. 2007. Practical testing of a C99 compiler using output comparison. Software: Practice & Experience (SPE) 37, 14 (Nov. 2007), 1475--1488.
[53]
Steve Sims, Rance Cleaveland, Ken Butts, and Scott Ranville. 2001. Automated validation of software models. In Proc. 16th International Conference on Automated Software Engineering (ASE). IEEE, 91--96.
[54]
Adepu Sridhar, D. Srinivasulu, and Durga Prasad Mohapatra. 2013. Model-based test-case generation for Simulink/Stateflow using dependency graph approach. In Proc. 3rd IEEE International Advance Computing Conference (IACC). 1414--1419.
[55]
Matthew Stephan, Manar H. Alalfi, and James R. Cordy. 2014. Towardsa taxonomy for Simulink model mutations. In Proc. 7th IEEE International Conference on Software Testing, Verification and Validation (ICST) Workshops. IEEE, 206--215.
[56]
Ingo Stürmer and Mirko Conrad. 2003. Test suite design for code generation tools. In Proc. 18th IEEE International Conference on Automated Software Engineering (ASE). 286--290.
[57]
Ingo Stürmer, Mirko Conrad, Heiko Dörr, and Peter Pepper. 2007. Systematic Testing of Model-Based Code Generators. IEEE Transactions on Software Engineering (TSE) 33, 9 (Sept. 2007), 622--634.
[58]
Zhendong Su and Chengnian Sun. 2020. EMI Compiler Validation Project. http://web.cs.ucdavis.edu/~su/emi-project/. Accessed Jan 2020.
[59]
Chengnian Sun, Vu Le, and Zhendong Su. 2016. Finding Compiler Bugs via Live Code Mutation. SIGPLAN Not. 51, 10 (Oct. 2016), 849--863.
[60]
Qiuming Tao, Wei Wu, Chen Zhao, and Wuwei Shen. 2010. An Automatic Testing Approach for Compiler Based on Metamorphic Testing Technique. In Proc. 2010 Asia Pacific Software Engineering Conference (APSEC). IEEE, 270--279.
[61]
Quang Minh Tran, Benjamin Wilmes, and Christian Dziobek. 2013. Refactoring of Simulink diagrams via composition of transformation steps. In Proc. 8th International Conference on Software Engineering Advances (ICSEA). IARIA, 140--145.
[62]
Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. 2011. Finding and understanding bugs in C compilers. In Proc. 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). ACM, 283--294.
[63]
Justyna Zander, Ina Schieferdecker, and Pieter J. Mosterman. 2011. Model-based testing for embedded systems (first ed.). CRC Press.
[64]
Yuan Zhan and John A. Clark. 2005. Search-based mutation testing for Simulink models. In Proc. Genetic and Evolutionary Computation Conference (GECCO). ACM, 1061--1068.
[65]
Liang Zou, Naijun Zhan, Shuling Wang, and Martin Fränzle. 2015. Formal Verification of Simulink/Stateflow Diagrams. In Proc. 13th International Symposium on Automated Technology for Verification and Analysis (ATVA). Springer, 464--481.

Cited By

View all
  • (2024)Enumerating Valid Non-Alpha-Equivalent Programs for Interpreter TestingACM Transactions on Software Engineering and Methodology10.1145/364799433:5(1-31)Online publication date: 4-Jun-2024
  • (2024)Metamorphic Testing of Secure Multi-party Computation (MPC) CompilersProceedings of the ACM on Software Engineering10.1145/36437811:FSE(1216-1237)Online publication date: 12-Jul-2024
  • (2024)How Does Simulation-Based Testing for Self-Driving Cars Match Human Perception?Proceedings of the ACM on Software Engineering10.1145/36437681:FSE(929-950)Online publication date: 12-Jul-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICSE '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering
June 2020
1640 pages
ISBN:9781450371216
DOI:10.1145/3377811
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

In-Cooperation

  • KIISE: Korean Institute of Information Scientists and Engineers
  • IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 October 2020

Permissions

Request permissions for this article.

Check for updates

Badges

Author Tags

  1. cyber-physical systems
  2. differential testing
  3. equivalence modulo input
  4. model mutation
  5. simulink

Qualifiers

  • Research-article

Funding Sources

Conference

ICSE '20
Sponsor:

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)166
  • Downloads (Last 6 weeks)22
Reflects downloads up to 23 Dec 2024

Other Metrics

Citations

Cited By

View all
  • (2024)Enumerating Valid Non-Alpha-Equivalent Programs for Interpreter TestingACM Transactions on Software Engineering and Methodology10.1145/364799433:5(1-31)Online publication date: 4-Jun-2024
  • (2024)Metamorphic Testing of Secure Multi-party Computation (MPC) CompilersProceedings of the ACM on Software Engineering10.1145/36437811:FSE(1216-1237)Online publication date: 12-Jul-2024
  • (2024)How Does Simulation-Based Testing for Self-Driving Cars Match Human Perception?Proceedings of the ACM on Software Engineering10.1145/36437681:FSE(929-950)Online publication date: 12-Jul-2024
  • (2024)Agile Field Test Support for Drone Rescue Missions2024 26th International Conference on Business Informatics (CBI)10.1109/CBI62504.2024.00034(228-237)Online publication date: 9-Sep-2024
  • (2023)A Generative and Mutational Approach for Synthesizing Bug-Exposing Test Cases to Guide Compiler FuzzingProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616332(1127-1139)Online publication date: 30-Nov-2023
  • (2023)Harnessing Large Language Models for Simulink Toolchain Testing and Developing Diverse Open-Source Corpora of Simulink Models for Metric and Evolution AnalysisProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3605233(1541-1545)Online publication date: 12-Jul-2023
  • (2023)Automated Identification and Qualitative Characterization of Safety Concerns Reported in UAV Software PlatformsACM Transactions on Software Engineering and Methodology10.1145/356482132:3(1-37)Online publication date: 26-Apr-2023
  • (2023)DeepDiffer: Find Deep Learning Compiler Bugs via Priority-guided Differential Fuzzing2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security (QRS)10.1109/QRS60937.2023.00066(616-627)Online publication date: 22-Oct-2023
  • (2023)EvoSL: A Large Open-Source Corpus of Changes in Simulink Models & Projects2023 ACM/IEEE 26th International Conference on Model Driven Engineering Languages and Systems (MODELS)10.1109/MODELS58315.2023.00024(273-284)Online publication date: 1-Oct-2023
  • (2023)ScoutSL: An Open-Source Simulink Search Engine2023 ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C)10.1109/MODELS-C59198.2023.00022(70-74)Online publication date: 1-Oct-2023
  • Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media