research-article

Tackling the equivalent mutant problem in real-time systems: the 12 commandments of model-based mutation testing

Authors:

Maurice H. ter Beek,

Maxime Cordy, and

Axel LegayAuthors Info & Claims

SPLC '20: Proceedings of the 24th ACM Conference on Systems and Software Product Line: Volume A - Volume A

October 2020

Article No.: 30, Pages 1 - 11

https://doi.org/10.1145/3382025.3414966

Published: 19 October 2020 Publication History

Abstract

Mutation testing can effectively drive test generation to reveal faults in software systems. However, it faces a typical efficiency issue as it can produce many mutants that are equivalent to the original system, making it impossible to generate test cases from them.

We consider this problem when model-based mutation testing is applied to real-time system product lines, represented as timed automata. We define novel, time-specific mutation operators and formulate the equivalent mutant problem in the frame of timed refinement relations.

Further, we study in which cases a mutation yields an equivalent mutant. Our theoretical results provide guidance to system engineers, allowing them to eliminate mutations from which no test case can be produced. Our evaluation, based on a proof-of-concept tool and an industrial case from the automotive domain, confirms the validity of our theory and demonstrates that our approach can eliminate many of the equivalent mutants (88% in our case study).

References

[1]

Bernhard K. Aichernig, Harald Brandl, Elisabeth Jöbstl, Willibald Krenn, Rupert Schlick, and Stefan Tiran. 2015. Killing Strategies for Model-Based Mutation Testing. Softw. Test. Verif. Reliab. 25, 8 (2015), 716--748.

Digital Library

[2]

Bernhard K. Aichernig, Florian Lorber, and Dejan Nickovic. 2013. Time for Mutants: Model-Based Mutation Testing with Timed Automata. In Proceedings of the 7th International Conference on Tests and Proofs (TAP'13) (LNCS, Vol. 7942), Margus Veanes and Luca Viganò (Eds.). Springer, 20--38.

[3]

Rajeev Alur and David L. Dill. 1994. A Theory of Timed Automata. Theoret. Comput. Sci. 126, 2 (1994), 183--235.

Digital Library

[4]

James H. Andrews, Lionel C. Briand, Yvan Labiche, and Akbar S. Namin. 2006. Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria. IEEE Trans. Softw. Eng. 32, 8 (2006), 608--624.

Digital Library

[5]

Eugene Asarin, Oded Maler, Amir Pnueli, and Joseph Sifakis. 1998. Controller Synthesis for Timed Automata. IFAC Proc. Vol. 31, 18 (1998), 447--452. Proceedings of the 5th IFAC Conference on System Structure and Control (SSC'98).

[6]

Richard Baker and Ibrahim Habli. 2013. An Empirical Evaluation of Mutation Testing for Improving the Test Quality of Safety-Critical Software. IEEE Trans. Softw. Eng. 39, 6 (2013), 787--805.

Digital Library

[7]

Gerd Behrmann, Agnès Cougnard, Alexandre David, Emmanuel Fleury, Kim G. Larsen, and Didier Lime. 2007. UPPAAL-Tiga: Time for Playing Games!. In Proceedings of the 19th International Conference on Computer Aided Verification (CAV'07) (LNCS, Vol. 4590), Werner Damm and Holger Hermanns (Eds.). Springer, 121--125.

[8]

Angelo Brillout, Nannan He, Michele Mazzucchi, Daniel Kroening, Mitra Purandare, Philipp Rümmer, and Georg Weissenbacher. 2009. Mutation-Based Test Case Generation for Simulink Models. In Proceedings of the 8th International Symposium on Formal Methods for Components and Objects (FMCO'09) (LNCS, Vol. 6286), Frank S. de Boer, Marcello M. Bonsangue, Stefan Hallerstede, and Michael Leuschel (Eds.). Springer, 208--227.

[9]

Tsun S. Chow. 1978. Testing Software Design Modeled by Finite-State Machines. IEEE Trans. Softw. Eng. SE-4, 3 (1978), 178--187.

Digital Library

[10]

Andreas Classen, Maxime Cordy, Pierre-Yves Schobbens, Patrick Heymans, Axel Legay, and Jean-François Raskin. 2013. Featured Transition Systems: Foundations for Verifying Variability-Intensive Systems and Their Application to LTL Model Checking. IEEE Trans. Softw. Eng. 39, 8 (2013), 1069--1089.

Digital Library

[11]

Maxime Cordy, Andreas Classen, Gilles Perrouin, Pierre-Yves Schobbens, Patrick Heymans, and Axel Legay. 2012. Simulation-based abstractions for software product-line model checking. In 34th International Conference on Software Engineering, ICSE 2012, June 2--9, 2012, Zurich, Switzerland, Martin Glinz, Gail C. Murphy, and Mauro Pezzè (Eds.). IEEE Computer Society, 672--682.

[12]

Maxime Cordy, Axel Legay, Pierre-Yves Schobbens, and Louis-Marie Traonouez. 2013. A Framework for the Rigorous Design of Highly Adaptive Timed Systems. In Proceedings of the 1st FME Workshop on Formal Methods in Software Engineering (FormaliSE'13). IEEE, 64--70.

[13]

Maxime Cordy, Pierre-Yves Schobbens, Patrick Heymans, and Axel Legay. 2012. Behavioural Modelling and Verification of Real-Time Software Product Lines. In Proceedings of the 16th International Software Product Line Conference (SPLC'12). ACM, 66--75.

Digital Library

[14]

Alexandre David, Kim G. Larsen, Axel Legay, Ulrik Nyman, and Andrzej Wąsowski. 2010. Timed I/O Automata: A Complete Specification Theory for Realtime Systems. In Proceedings of the 13th International Conference on Hybrid Systems: Computation and Control (HSCC'10). ACM, 91--100.

Digital Library

[15]

Alexandre David, Kim Guldstrand Larsen, Axel Legay, Ulrik Nyman, and Andrzej Wąsowski. 2010. ECDAR: An Environment for Compositional Design and Analysis of Real Time Systems. In Proceedings of the 8th International Symposium on Automated Technology for Verification and Analysis (ATVA'10) (LNCS, Vol. 6252), Ahmed BouajjaniWei-Ngan Chin (Ed.). Springer, 365--370.

[16]

Xavier Devroey, Gilles Perrouin, Mike Papadakis, Axel Legay, Pierre-Yves Schobbens, and Patrick Heymans. 2016. Featured Model-based Mutation Analysis. In Proceedings of the 38th International Conference on Software Engineering (ICSE'16). ACM, 655--666.

Digital Library

[17]

Sandra Fabbri, José C. Maldonado, Tatiana Sugeta, and Paulo C. Masiero. 1999. Mutation testing applied to validate specifications based on statecharts. In Proceedings of the 10th International Symposium on Software Reliability Engineering (ISSRE'99). IEEE, 210--219.

[18]

Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Trans. Softw. Eng. 37, 5 (2011), 649--678.

Digital Library

[19]

Kim G. Larsen, Florian Lorber, Brian Nielsen, and Ulrik M. Nyman. 2017. Mutation-Based Test-Case Generation with Ecdar. In Proceedings of the 10th IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW'17). IEEE, 319--328.

[20]

Kim G. Larsen, Ulrik Nyman, and Andrzej Wąsowski. 2007. Modal I/O Automata for Interface and Product Line Theories. In Proceedings of the 16th European Symposium on Programming (ESOP'07) (LNCS, Vol. 4421), Rocco De Nicola (Ed.). Springer, 64--79.

[21]

Lech Madeyski, Wojciech Orzeszyna, Richard Torkar, and Mariusz Józala. 2014. Overcoming the Equivalent Mutant Problem: A Systematic Literature Review and a Comparative Experiment of Second Order Mutation. IEEE Trans. Softw. Eng. 40, 1 (2014), 23--42.

Digital Library

[22]

Jeff Offutt. 2011. A mutation carol: Past, present and future. Inf. Softw. Technol. 53, 10 (2011), 1098--1107.

[23]

Mike Papadakis and Nicos Malevris. 2010. An Empirical Evaluation of the First and Second Order Mutation Testing Strategies. In Proceedings of the 3rd International Conference on Software Testing, Verification and Validation Workshops (ICSTW'10). 90--99.

Digital Library

[24]

Maurice H. ter Beek, Sjef van Loo, Erik P. de Vink, and Tim A.C. Willemse. 2020. Family-Based SPL Model Checking Using Parity Games with Variability. In Proceedings of the 23rd International Conference on Fundamental Approaches to Software Engineering (FASE'20) (LNCS, Vol. 12076), Heike Wehrheim and Jordi Cabot (Eds.). Springer, 245--265.

[25]

Mark Utting, Alexander Pretschner, and Bruno Legeard. 2012. A Taxonomy of Model-Based Testing Approaches. Softw. Test. Verif. Reliab. 22, 5 (2012), 297--312.

Digital Library

[26]

Elaine Weyuker, Tarak Goradia, and Ashutosh Singh. 1994. Automatically Generating Test Data from a Boolean Specification. IEEE Trans. Softw. Eng. 20, 5 (1994), 353--363.

Digital Library

Cited By

Cortés DOrtiz JBasile DAranda JPerrouin GSchobbens PFuria CLopes APlat NGnesi S(2024)Time for Networks: Mutation Testing for Timed Automata NetworksProceedings of the 2024 IEEE/ACM 12th International Conference on Formal Methods in Software Engineering (FormaliSE)10.1145/3644033.3644378(44-54)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3644033.3644378
Betancourt JOrtiz JAranda J(2023)Applying parallelism to a bisimulation algorithm to improve efficiency in software testing of time-critical systemsIngeniería y Competitividad10.25100/iyc.v25iSuplemento.1314425:SuplementoOnline publication date: 29-Nov-2023
https://doi.org/10.25100/iyc.v25iSuplemento.13144
Cuartas JAranda JCordy MOrtiz JPerrouin GSchobbens P(2023)MUPPAAL: Reducing and Removing Equivalent and Duplicate Mutants in UPPAAL2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW58534.2023.00021(52-61)Online publication date: Apr-2023
https://doi.org/10.1109/ICSTW58534.2023.00021
Show More Cited By

Index Terms

Tackling the equivalent mutant problem in real-time systems: the 12 commandments of model-based mutation testing
1. Software and its engineering
  1. Software creation and management
    1. Software development techniques
      1. Reusability
        Software product lines
    2. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software organization and properties
    1. Software system structures
      1. Real-time systems software

Recommendations

Static detection of equivalent mutants in real-time model-based mutation testing: An Empirical Evaluation
Abstract
Model-based mutation testing has the potential to effectively drive test generation to reveal faults in software systems. However, it faces a typical efficiency issue since it could produce many mutants that are equivalent to the original system ...
Read More
Conformance testing of real-time component based systems
ISSADS'05: Proceedings of the 5th international conference on Advanced Distributed Systems

In this paper, we suggest a methodology for testing Real Time Component Based Systems (RTCBS). A RTCBS is described as a collection of components where each component is modeled as a Timed Input-Output Automaton (TIOA). The first part of this study is ...
Read More
A new method for testing real time systems
RTCSA '00: Proceedings of the Seventh International Conference on Real-Time Systems and Applications

In this article, we present a framework for testing real-time systems. More precisely, (1) we propose a method for generating test sequences, (2) we determine constraints which guarantee the executability of test sequences, and (3) we propose a ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SPLC '20: Proceedings of the 24th ACM Conference on Systems and Software Product Line: Volume A - Volume A

October 2020

323 pages

ISBN:9781450375696

DOI:10.1145/3382025

General Chair:
Roberto Erick Lopez-Herrejon
École de technologie supérieure, Canada

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SPLC '20

Sponsor:

SIGSOFT

SPLC '20: 24th ACM International Systems and Software Product Line Conference

October 19 - 23, 2020

Quebec, Montreal, Canada

Acceptance Rates

SPLC '20 Paper Acceptance Rate 17 of 49 submissions, 35%;

Overall Acceptance Rate 167 of 463 submissions, 36%

Upcoming Conference

SPLC '24

Sponsor:
sigsoft

28th ACM International Systems and Software Product Line Conference

September 2 - 6, 2024

Dommeldange , Luxembourg

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
104
Total Downloads

Downloads (Last 12 months)14
Downloads (Last 6 weeks)1

Other Metrics

View Author Metrics

Citations

Cited By

Cortés DOrtiz JBasile DAranda JPerrouin GSchobbens PFuria CLopes APlat NGnesi S(2024)Time for Networks: Mutation Testing for Timed Automata NetworksProceedings of the 2024 IEEE/ACM 12th International Conference on Formal Methods in Software Engineering (FormaliSE)10.1145/3644033.3644378(44-54)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3644033.3644378
Betancourt JOrtiz JAranda J(2023)Applying parallelism to a bisimulation algorithm to improve efficiency in software testing of time-critical systemsIngeniería y Competitividad10.25100/iyc.v25iSuplemento.1314425:SuplementoOnline publication date: 29-Nov-2023
https://doi.org/10.25100/iyc.v25iSuplemento.13144
Cuartas JAranda JCordy MOrtiz JPerrouin GSchobbens P(2023)MUPPAAL: Reducing and Removing Equivalent and Duplicate Mutants in UPPAAL2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW58534.2023.00021(52-61)Online publication date: Apr-2023
https://doi.org/10.1109/ICSTW58534.2023.00021
Yik LNasir bin Wan Kadir WIbrahim N(2023)A Systematic Literature Review on Solutions of Mutation Testing Problems2023 IEEE 8th International Conference On Software Engineering and Computer Systems (ICSECS)10.1109/ICSECS58457.2023.10256324(64-71)Online publication date: 25-Aug-2023
https://doi.org/10.1109/ICSECS58457.2023.10256324
Basile DTer Beek MGöttmann HLochau M(2023)Mutant Equivalence as Monotonicity in Parametric Timed Games2023 IEEE/ACM 11th International Conference on Formal Methods in Software Engineering (FormaliSE)10.1109/FormaliSE58978.2023.00014(55-65)Online publication date: May-2023
https://doi.org/10.1109/FormaliSE58978.2023.00014
Sidorenko GMostowski WVinel ASjoberg JCooney M(2021)The CAR Approach: Creative Applied Research Experiences for Master’s Students in Autonomous Platooning2021 30th IEEE International Conference on Robot & Human Interactive Communication (RO-MAN)10.1109/RO-MAN50785.2021.9515560(214-221)Online publication date: 8-Aug-2021
https://doi.org/10.1109/RO-MAN50785.2021.9515560

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents