research-article

Mutation testing strategies using mutant classification

Authors:

Mike Papadakis,

Yves Le TraonAuthors Info & Claims

SAC '13: Proceedings of the 28th Annual ACM Symposium on Applied Computing

Pages 1223 - 1229

https://doi.org/10.1145/2480362.2480592

Published: 18 March 2013 Publication History

Abstract

Mutation testing has a widespread reputation of being a rather powerful testing technique. However, its practical application requires the detection of equivalent mutants. Detecting equivalent mutants is cumbersome since it requires manual analysis, resulting in unbearable testing cost. To overcome this difficulty, researchers have proposed the use of mutant classification, an approach that helps isolating equivalent mutants. From this perspective, the present paper establishes and assesses possible mutant classification strategies. The conducted study suggests that while mutant classification can be useful in isolating equivalent mutants, it fails to kill some mutants. Indeed, the experimental results show that the proposed strategies achieve to kill approximately 95% of the introduced killable mutants.

References

[1]

Adamopoulos, K., Harman, M. and Hierons, R. M. 2004. How to Overcome the Equivalent Mutant Problem and Achieve Tailored Selective Mutation Using Co-evolution. In GECCO 2004, 1338--1349.

[2]

Agrawal, H., DeMillo, R. A., Hathaway, B., Hsu, W., Hsu, W., Krauser, E. W., Martin, R. J., Mathur, A. P. and Spafford, E. 1989. Design of Mutant Operators for the C Programming Language. Purdue University.

[3]

Andrews, J. H., Briand, L. C. and Labiche, Y. 2005. Is mutation an appropriate tool for testing experiments? In Proceedings of the 27th international conference on Software engineering (ICSE '05), 402--411.

Digital Library

[4]

Budd, T. A. and Angluin, D. 1982. Two notions of correctness and their relation to testing. Acta Informatica. 18, 1 (1982), 31--45.

Digital Library

[5]

DeMillo, R. A., Lipton, R. J. and Sayward, F. G. 1978. Hints on Test Data Selection: Help for the Practicing Programmer. Computer. 11, 4 (1978), 34--41.

Digital Library

[6]

Delamaro, M. and Maldonado, J. C. 1996. Proteum - A Tool for the Assessment of Test Adequacy for C Programs. In Proceedings of the Conference on Performability in Computing Systems (PCS '96), 79--95.

[7]

Do, H., Elbaum, S. and Rothermel, G. 2005. Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact. Empirical Softw. Engg. 10, 4 (2005), 405--435.

Digital Library

[8]

Harder, M., Mellen, J. and Ernst, M. D. 2003. Improving test suites via operational abstraction. In Proceedings of the 25th International Conference on Software Engineering (ICSE '03), 60--71.

Digital Library

[9]

Hierons, R. M., Harman, M. and Danicic, S. 1999. Using Program Slicing to Assist in the Detection of Equivalent Mutants. Software Testing, Verification and Reliability. 9, 4 (1999), 233--262.

[10]

Hutchins, M., Foster, H., Goradia, T. and Ostrand, T. 1994. Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria. In Proceedings of the 16th international conference on Software engineering (ICSE '94) 191--200.

Digital Library

[11]

Jia, Y. and Harman, M. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Trans. Softw. Eng. 37, 5 (September 2011), 649--678.

Digital Library

[12]

Jia, Y. and Harman, M. 2009. Higher Order Mutation Testing. Inf. Softw. Technol. 51, 10 (2009), 1379--1393.

Digital Library

[13]

Kintis, M., Papadakis, M. and Malevris, N. 2010. Evaluating Mutation Testing Alternatives: A Collateral Experiment. In Proceedings of the 2010 Asia Pacific Software Engineering Conference (APSEC '10), 300--309.

Digital Library

[14]

Kintis, M., Papadakis, M. and Malevris, N. 2012. Isolating First Order Equivalent Mutants via Second Order Mutation. In Proceedings of the 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation (ICST '12), 701--710.

Digital Library

[15]

Namin, A. S., Andrews, J. H. and Murdoch, D. J. 2008. Sufficient mutation operators for measuring test effectiveness. In Proceedings of the 30th international conference on Software engineering (ICSE '08), 351--360.

Digital Library

[16]

Offutt, A. J. and Pan, J. 1997. Automatically Detecting Equivalent Mutants and Infeasible Paths. Software Testing, Verification and Reliability. 7, 3 (1997), 165--192.

[17]

Offutt, A. J. and Untch, R. H. 2001. Mutation 2000: uniting the orthogonal. In Mutation testing for the new century. Kluwer Academic Publishers, 34--44.

Digital Library

[18]

Papadakis, M. and Malevris, N. 2010. An Empirical Evaluation of the First and Second Order Mutation Testing Strategies. In Proceedings of the 2010 Third International Conference on Software Testing, Verification, and Validation Workshops (ICSTW '10), 90--99.

Digital Library

[19]

Papadakis, M. and Malevris, N. 2010. Automatic Mutation Test Case Generation via Dynamic Symbolic Execution. In Proceedings of the 2010 IEEE 21st International Symposium on Software Reliability Engineering (ISSRE '10), 121--130.

Digital Library

[20]

Papadakis, M. and Malevris, N. 2011. Automatically performing weak mutation with the aid of symbolic execution, concolic testing and search-based testing. Software Quality Journal. 19, 4 (2011), 691--723.

Digital Library

[21]

Schuler, D., Dallmeier, V. and Zeller, A. 2009. Efficient mutation testing by checking invariant violations. In Proceedings of the eighteenth international symposium on Software testing and analysis. (ISSTA '09), 69--80.

Digital Library

[22]

Schuler, D. and Zeller, A. 2012. Covering and Uncovering Equivalent Mutants. Software Testing, Verification and Reliability. (2012).

Cited By

Gong DWang TSu XZhang Y(2022)Equivalent Mutants Detection Based on Weighted Software Behavior GraphInternational Journal of Software Engineering and Knowledge Engineering10.1142/S021819402250032232:06(819-843)Online publication date: 24-Jun-2022
https://doi.org/10.1142/S0218194022500322
Dang XGong DYao XTian TLiu H(2022)Enhancement of Mutation Testing via Fuzzy Clustering and Multi-Population Genetic AlgorithmIEEE Transactions on Software Engineering10.1109/TSE.2021.305298748:6(2141-2156)Online publication date: 1-Jun-2022
https://doi.org/10.1109/TSE.2021.3052987
Yao XZhang GPan FGong DWei C(2022)Orderly Generation of Test Data via Sorting Mutant Branches Based on Their Dominance Degrees for Weak Mutation TestingIEEE Transactions on Software Engineering10.1109/TSE.2020.301496048:4(1169-1184)Online publication date: 1-Apr-2022
https://doi.org/10.1109/TSE.2020.3014960
Show More Cited By

Index Terms

Mutation testing strategies using mutant classification
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Prioritizing mutants to guide mutation testing
ICSE '22: Proceedings of the 44th International Conference on Software Engineering

Mutation testing offers concrete test goals (mutants) and a rigorous test efficacy criterion, but it is expensive due to vast numbers of mutants, many of which are neither useful nor actionable. Prior work has focused on selecting representative and ...
Predictive mutation testing
ISSTA 2016: Proceedings of the 25th International Symposium on Software Testing and Analysis

Mutation testing is a powerful methodology for evaluating test suite quality. In mutation testing, a large number of mutants are generated and executed against the test suite to check the ratio of killed mutants. Therefore, mutation testing is widely ...
Faster mutation testing inspired by test prioritization and reduction
ISSTA 2013: Proceedings of the 2013 International Symposium on Software Testing and Analysis

Mutation testing is a well-known but costly approach for determining test adequacy. The central idea behind the approach is to generate mutants, which are small syntactic transformations of the program under test, and then to measure for a given test ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SAC '13: Proceedings of the 28th Annual ACM Symposium on Applied Computing

March 2013

2124 pages

ISBN:9781450316569

DOI:10.1145/2480362

Conference Chairs:
Sung Y. Shin
South Dakota State University, United States
,
José Carlos Maldonado
ICMC - University of São Paulo, Brazil

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGAPP: ACM Special Interest Group on Applied Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 March 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SAC '13

Sponsor:

SIGAPP

SAC '13: SAC '13

March 18 - 22, 2013

Coimbra, Portugal

Acceptance Rates

SAC '13 Paper Acceptance Rate 255 of 1,063 submissions, 24%;

Overall Acceptance Rate 1,650 of 6,669 submissions, 25%

Upcoming Conference

SAC '25

Sponsor:
sigapp

The 40th ACM/SIGAPP Symposium on Applied Computing

March 31 - April 4, 2025

Catania , Italy

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
291
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Gong DWang TSu XZhang Y(2022)Equivalent Mutants Detection Based on Weighted Software Behavior GraphInternational Journal of Software Engineering and Knowledge Engineering10.1142/S021819402250032232:06(819-843)Online publication date: 24-Jun-2022
https://doi.org/10.1142/S0218194022500322
Dang XGong DYao XTian TLiu H(2022)Enhancement of Mutation Testing via Fuzzy Clustering and Multi-Population Genetic AlgorithmIEEE Transactions on Software Engineering10.1109/TSE.2021.305298748:6(2141-2156)Online publication date: 1-Jun-2022
https://doi.org/10.1109/TSE.2021.3052987
Yao XZhang GPan FGong DWei C(2022)Orderly Generation of Test Data via Sorting Mutant Branches Based on Their Dominance Degrees for Weak Mutation TestingIEEE Transactions on Software Engineering10.1109/TSE.2020.301496048:4(1169-1184)Online publication date: 1-Apr-2022
https://doi.org/10.1109/TSE.2020.3014960
Dang XYao XGong DTian TSun B(2020)Multi-Task Optimization-Based Test Data Generation for Mutation Testing via Relevance of Mutant Branch and Input VariableIEEE Access10.1109/ACCESS.2020.30142908(144401-144412)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.3014290
Cutigi Ferrari FViola Pizzoleto AOffutt J(2018)A Systematic Review of Cost Reduction Techniques for Mutation Testing: Preliminary Results2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW.2018.00021(1-10)Online publication date: Apr-2018
https://doi.org/10.1109/ICSTW.2018.00021
Papadakis MDelamaro MLe Traon Y(2018)Mitigating the effects of equivalent mutants with mutant classification strategiesScience of Computer Programming10.1016/j.scico.2014.05.01295:P3(298-319)Online publication date: 31-Dec-2018
https://dl.acm.org/doi/10.1016/j.scico.2014.05.012
Houshmand MPaydar S(2017)TCE+: An Extension of the TCE Method for Detecting Equivalent Mutants in Java ProgramsFundamentals of Software Engineering10.1007/978-3-319-68972-2_11(164-179)Online publication date: 11-Oct-2017
https://doi.org/10.1007/978-3-319-68972-2_11
Bhatia VSinghal A(2016)Design of a Fuzzy model to detect equivalent mutants for weak and strong mutation testing2016 International Conference on Information Technology (InCITe) - The Next Generation IT Summit on the Theme - Internet of Things: Connect your Worlds10.1109/INCITE.2016.7857578(1-6)Online publication date: Oct-2016
https://doi.org/10.1109/INCITE.2016.7857578

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents