Test Case Generation for Simulink Models using Model Fuzzing and State Solving
Authors: 
Zhuo Su, Zehong Yu, Dongyan Wang, Wanli Chang, Bin Gu, Yu JiangAuthors Info & Claims
Zhuo Su, Zehong Yu, Dongyan Wang, Wanli Chang, Bin Gu, Yu JiangAuthors Info & ClaimsAbstract
Simulink plays an important role in the industry for modeling and synthesis of embedded systems. Ensuring system stability requires using numerous test cases to validate the functionality and safety of the models. However, as requirements increase, the complexity of the models poses new challenges to traditional testing methods. Traditional methods such as constraint solving and random search run into significant obstacles when navigating the complex branching logic and states within models.
In this paper, we introduce HybridTCG, a test case generation method by collaborating model fuzzing and state solving for Simulink models. First, HybridTCG starts a code-based fuzzer to generate high-coverage test cases rapidly. Then, it refines the test cases generated by the fuzzer, preserving only those that can achieve new model coverage. These selected test cases are input into the state-solving engine to derive corresponding states and resolve the constraints of subsequent branches. Ultimately, the test cases produced by the solving engine will be fed back into the fuzzer as high-quality seeds to enhance the fuzzing process. We have implemented HybridTCG and conducted a comprehensive evaluation using various benchmark Simulink models. Compared to the built-in Simulink Design Verifier and state-of-the-art academic work SimCoTest and STCG, HybridTCG achieves an average improvement of 54%, 108% and 24% on Decision Coverage, 50%, 62% and 6% on Condition Coverage, 291%, 282% and 45% on Modified Condition Decision Coverage, respectively. Moreover, HybridTCG is also much more efficient in testing than other tools.
References
[1]
William Aldrich. 2002. Using model coverage analysis to improve the controls development process. In AIAA Modeling and Simulation Technologies Conference and Exhibit. 4684--4694.
[2]
Saswat Anand, Edmund K Burke, Tsong Yueh Chen, John Clark, Myra B Cohen, Wolfgang Grieskamp, Mark Harman, Mary Jean Harrold, Phil McMinn, Antonia Bertolino, et al. 2013. An orchestrated survey of methodologies for automated software test case generation. Journal of Systems and Software 86, 8 (2013), 1978--2001.
[3]
Aitor Arrieta, Shuai Wang, Urtzi Markiegi, Goiuria Sagardui, and Leire Etxeberria. 2017. Search-based test case generation for cyber-physical systems. In 2017 IEEE Congress on Evolutionary Computation. 688--697.
[4]
Tansu Zafer Asici, Burak Karaduman, Raheleh Eslampanah, Moharram Challenger, Joachim Denil, and Hans Vangheluwe. 2019. Applying model driven engineering techniques to the development of contiki-based IoT systems. In Proceedings of the 1st International Workshop on Software Engineering Research & Practices for the Internet of Things. IEEE Press, 25--32.
[5]
Ezio Bartocci, Niveditha Manjunath, Leonardo Mariani, Cristinel Mateis, and Dejan Ničković. 2021. CPSDebug: Automatic failure explanation in CPS models. International Journal on Software Tools for Technology Transfer (2021), 1--14.
[6]
Axel Belinfante, Lars Frantzen, and Christian Schallhart. 2005. 14 tools for test case generation. In Model-based testing of reactive systems. Springer, 391--438.
[7]
Cristian Cadar, Daniel Dunbar, and Dawson R. Engler. 2008. KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs. In 8th USENIX Symposium on Operating Systems Design and Implementation, December 8--10, San Diego, California, USA. USENIX Association, 209--224.
[8]
Yaohui Chen, Mansour Ahmadi, Reza Mirzazade Farkhani, Boyu Wang, and Long Lu. 2020. MEUZZ: Smart Seed Scheduling for Hybrid Fuzzing. In 23rd International Symposium on Research in Attacks, Intrusions and Defenses, San Sebastian, Spain, October 14--15. USENIX Association, 77--92.
[9]
Edmund Clarke, Daniel Kroening, and Flavio Lerda. 2004. A Tool for Checking ANSI-C Programs. In Tools and Algorithms for the Construction and Analysis of Systems (Lecture Notes in Computer Science, Vol. 2988), Kurt Jensen and Andreas Podelski (Eds.). Springer, 168--176.
[10]
Rance Cleaveland, Scott A Smolka, and Steven T Sims. 2006. An Anstrumentation-Based Approach to Controller Model Validation. In Automotive Software Workshop. Springer, 84--97.
[11]
Jean-Louis Colaco, Bruno Pagano, and Marc Pouzet. 2017. SCADE 6: A formal language for embedded critical software development. In 11th International Symposium on Theoretical Aspects of Software Engineering. 1--11.
[12]
Frank Elberzhager, Alla Rosbach, and Thomas Bauer. 2013. Analysis and testing of matlab simulink models: a systematic mapping study. In Proceedings of the 2013 International Workshop on Joining AcadeMiA and Industry Contributions to Testing Automation. 29--34.
[13]
Google. 2016. LibFuzzer Tutorial. https://github.com/google/fuzzing/blob/master/tutorial/libFuzzerTutorial.md. Accessed: 2024-03-29.
[14]
Reinhard Hametner, Benjamin Kormann, Birgit Vogel-Heuser, Dietmar Winkler, and Alois Zoitl. 2011. Test case generation approach for industrial automation systems. In The 5th international conference on automation, robotics and applications. IEEE, 57--62.
[15]
Grégoire Hamon, Bruno Dutertre, Levent Erkok, John Matthews, Daniel Sheridan, David Cok, John Rushby, Peter Bokor, Sandeep Shukla, Andras Pataricza, et al. 2008. Simulink Design Verifier-Applying Automated Formal Methods to Simulink and Stateflow. In Third Workshop on Automated Formal Methods.
[16]
Nannan He, Philipp Rümmer, and Daniel Kroening. 2011. Test-Case Generation for Embedded Simulink via Formal Concept Analysis. In Proceedings of the 48th Design Automation Conference. 224--229.
[17]
Itti Hooda and Rajender Chhillar. 2014. A review: study of test case generation techniques. International Journal of Computer Applications 107, 16 (2014), 33--37.
[18]
SRI International. [n. d.]. Symbolic Analysis Laboratory. https://sal.csl.sri.com/
[19]
Karim Jahed and Juergen Dingel. 2019. Enabling model-driven software development tools for the internet of things. In Proceedings of the 11th International Workshop on Modelling in Software Engineerings. IEEE Press, 93--99.
[20]
Yu Jiang, Houbing Song, Yixiao Yang, Han Liu, Ming Gu, Yong Guan, Jiaguang Sun, and Lui Sha. 2018. Dependable model-driven development of cps: From stateflow simulation to verified implementation. ACM Transactions on Cyber-Physical Systems 3, 1 (2018), 12.
[21]
Philip Koopman and Tridib Chakravarty. 2004. Cyclic redundancy code (CRC) polynomial selection for embedded networks. In International Conference on Dependable Systems and Networks, 2004. IEEE, 145--154.
[22]
J Krizan, L Ertl, M Bradac, M Jasansky, and A Andreev. 2014. Automatic code generation from MATLAB/Simulink for critical applications. In 2014 IEEE 27th Canadian Conference on Electrical and Computer Engineering (CCECE). IEEE, 1--6.
[23]
Wenbin Li, Franck Le Gall, and Naum Spaseski. 2018. A survey on model-based testing tools for test case generation. In Tools and Methods of Program Analysis: 4th International Conference, TMPA 2017, Moscow, Russia, March 3--4, 2017, Revised Selected Papers 4. Springer, 77--89.
[24]
MathWorks. 2023. Simulink Coverage. https://www.mathworks.com/products/simulink-coverage.html. Accessed: 2024-03-29.
[25]
Reza Matinnejad, Shiva Nejati, Lionel C Briand, and Thomas Bruckmann. 2016. SimCoTest: A test suite generation tool for Simulink/Stateflow controllers. In Proceedings of the 38th International Conference on Software Engineering Companion. 585--588.
[26]
Reza Matinnejad, Shiva Nejati, Lionel C Briand, and Thomas Bruckmann. 2018. Test generation and test prioritization for simulink models with dynamic behavior. IEEE Transactions on Software Engineering 45, 9 (2018), 919--944.
[27]
Swarup Mohalik, Ambar A Gadkari, Anand Yeolekar, KC Shashidhar, and S Ramesh. 2014. Automatic test case generation from Simulink/Stateflow models using model checking. Software Testing, Verification and Reliability 24, 2 (2014), 155--180.
[28]
Cu D Nguyen, Alessandro Marchetto, and Paolo Tonella. 2012. Combining model-based and combinatorial testing for effective test case generation. In Proceedings of the 2012 International Symposium on Software Testing and Analysis. 100--110.
[29]
Jan Peleska, Elena Vorobev, and Florian Lapschies. 2011. Automated test case generation with SMT-solving and abstract interpretation. In NASA Formal Methods: Third International Symposium, NFM 2011, Pasadena, CA, USA, April 18-20, 2011. Proceedings 3. Springer, 298--312.
[30]
Florian Rademacher, Jonas Sorgalla, Sabine Sachweh, and Albert Zündorf. 2019. A model-driven workflow for distributed microservice development. In Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing. ACM, 1260--1262.
[31]
Manoranjan Satpathy, Anand Yeolekar, and S Ramesh. 2008. Randomized directed testing (REDIRECT) for Simulink/Stateflow models. In Proceedings of the 8th ACM international conference on Embedded software. 217--226.
[32]
Simulink and Matlab. [n. d.]. Simulink Documentation. https://www.mathworks.com/help/simulink/index.html
[33]
Sergey Staroletov, Nikolay Shilov, Vladimir Zyubin, Tatiana Liakh, Andrei Rozov, Ivan Konyukhov, Innokenty Shilov, Thomas Baar, and Horst Schulte. 2019. Model-driven methods to design of reliable multiagent cyber-physical systems. In Proc. of the Conference on Modeling and Analysis of Complex Systems and Processes.
[34]
Nick Stephens, John Grosen, Christopher Salls, Andrew Dutcher, Ruoyu Wang, Jacopo Corbetta, Yan Shoshitaishvili, Christopher Kruegel, and Giovanni Vigna. 2016. Driller: Augmenting Fuzzing Through Selective Symbolic Execution. In Annual Network and Distributed System Security Symposium.
[35]
Zhuo Su, Dongyan Wang, Yixiao Yang, Yu Jiang, Wanli Chang, Liming Fang, Wen Li, and Jiaguang Sun. 2021. Code Synthesis for Dataflow Based Embedded Software Design. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (2021).
[36]
Zhuo Su, Dongyan Wang, Zehong Yu, Yixiao Yang, Yu Jiang, Rui Wang, Wanli Chang, Wen Li, Aiguo Cui, and Jiaguang Sun. 2022. PHCG: Optimizing Simulink Code Generation for Embedded System with SIMD Instructions. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (2022).
[37]
Zhuo Su, Zehong Yu, Dongyan Wang, Yixiao Yang, Rui Wang, Wanli Chang, Aiguo Cui, and Yu Jiang. 2023. STCG: State-Aware Test Case Generation for Simulink Models. In 60th ACM/IEEE Design Automation Conference (DAC). ACM.
[38]
Mingzhe Wang, Jie Liang, Yuanliang Chen, Yu Jiang, Xun Jiao, Han Liu, Xibin Zhao, and Jiaguang Sun. 2018. SAFL: increasing and accelerating testing coverage with symbolic execution and guided fuzzing. In Proceedings of the 40th International Conference on Software Engineering: Companion ProceeedingsGothenburg, Sweden, May 27 - June 03. ACM, 61--64.
[39]
Zehong Yu, Zhuo Su, Yixiao Yang, Jie Liang, Yu Jiang, Aiguo Cui, Wanli Chang, and Rui Wang. 2022. Mercury: Instruction Pipeline Aware Code Generation for Simulink Models. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 41, 11 (2022), 4504--4515.
Index Terms
- Test Case Generation for Simulink Models using Model Fuzzing and State Solving
Index terms have been assigned to the content through auto-classification.
Recommendations
Automated test generation for OpenCL kernels using fuzzing and constraint solving
GPGPU '20: Proceedings of the 13th Annual Workshop on General Purpose Processing using Graphics Processing UnitGraphics Processing Units (GPUs) are massively parallel processors offering performance acceleration and energy efficiency unmatched by current processors (CPUs) in computers. These advantages along with recent advances in the programmability of GPUs ...
CFTCG: Test Case Generation for Simulink Model through Code Based Fuzzing
DAC '24: Proceedings of the 61st ACM/IEEE Design Automation ConferenceSimulink is extensively utilized in system design for its ability to facilitate modeling and synthesis of embedded controllers. It provides automatic test case generation to assist testers in inspecting the model. However, with the continuous increase in ...
Test-case generation for embedded simulink via formal concept analysis
DAC '11: Proceedings of the 48th Design Automation ConferenceMutation testing suffers from the high computational cost of automated test-vector generation, due to the large number of mutants that can be derived from programs and the cost of generating test-cases in a white-box manner. We propose a novel algorithm ...
Comments
Information & Contributors
Information
Published In
October 2024
2587 pages
ISBN:9798400712487
DOI:10.1145/3691620
- General Chair:
- Vladimir Filkov,
- Program Co-chairs:
- Baishakhi Ray,
- Minghui Zhou
Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 27 October 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
ASE '24: 39th IEEE/ACM International Conference on Automated Software Engineering
October 27 - November 1, 2024
CA, Sacramento, USA
Acceptance Rates
Overall Acceptance Rate 82 of 337 submissions, 24%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 108Total Downloads
- Downloads (Last 12 months)108
- Downloads (Last 6 weeks)37
Reflects downloads up to 08 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in