Article

Regression mutation testing

Authors:

Lingming Zhang,

Sarfraz KhurshidAuthors Info & Claims

ISSTA 2012: Proceedings of the 2012 International Symposium on Software Testing and Analysis

Pages 331 - 341

https://doi.org/10.1145/2338965.2336793

Published: 15 July 2012 Publication History

Abstract

Mutation testing is one of the most powerful approaches for evaluating quality of test suites. However, mutation testing is also one of the most expensive testing approaches. This paper presents Regression Mutation Testing (ReMT), a new technique to speed up mutation testing for evolving systems. The key novelty of ReMT is to incrementally calculate mutation testing results for the new program version based on the results from the old program version; ReMT uses a static analysis to check which results can be safely reused. ReMT also employs a mutation-specific test prioritization to further speed up mutation testing. We present an empirical study on six evolving systems, whose sizes range from 3.9KLoC to 88.8KLoC. The empirical results show that ReMT can substantially reduce mutation testing costs, indicating a promising future for applying mutation testing on evolving software systems.

References

[1]

PIT. http://pitest.org/.

[2]

A. T. Acree, T. A. Budd, R. A. DeMillo, R. J. Lipton, and F. G. Sayward. Mutation analysis. Technical report, Georgia Institute of Technology, 1979.

[3]

P. Ammann and J. Offutt. Introduction to Software Testing. Cambridge University Press.

[4]

J. H. Andrews, L. C. Briand, and Y. Labiche. Is mutation an appropriate tool for testing experiments? In Proc. ICSE, pages 402-411, 2005.

Digital Library

[5]

E. F. Barbosa, J. C. Maldonado, and A. M. R. Vincenzi. Toward the determination of sufficient mutant operators for C. STVR, 11(2):113-136, 2001.

[6]

P. K. Chittimalli and M. J. Harrold. Recomputing coverage information to assist regression testing. IEEE TSE, 35(4):452-469, 2009.

Digital Library

[7]

M. E. Delamaro and J. C. Maldonado. Proteum-A tool for the assessment of test adequacy for C programs. In Proc. the Conference on Performability in Computing Sys PCS 96, pages 79-95, 1996.

[8]

R. A. DeMillo, E.W. Krauser, and A. P. Mathur. Compiler-integrated program mutation. In Proc. COMPSAC, pages 351-356, 1991.

[9]

R. A. DeMillo, R. J. Lipton, and F. G. Sayward. Hints on test data selection: Help for the practicing programmer. Computer, 11(4):34-41, 1978.

Digital Library

[10]

R. A. DeMillo and R. J. Martin. The Mothra software testing environment user's manual. Technical report, Software Engineering Research Center, 1987.

[11]

R. A. DeMillo and A. J. Offutt. Constraint-based automatic test data generation. IEEE TSE, 17(9):900-910, 1991.

Digital Library

[12]

H. Do and G. Rothermel. On the use of mutation faults in empirical assessments of test case prioritization techniques. IEEE TSE, pages 733-752, 2006.

Digital Library

[13]

P. G. Frankl, S. N. Weiss, and C. Hu. All-uses vs mutation testing: An experimental comparison of effectiveness. JSS, 38(3):235-253, 1997.

Digital Library

[14]

G. Fraser and A. Zeller. Mutation-driven generation of unit tests and oracles. In ISSTA, pages 147-158, 2010.

Digital Library

[15]

R. G. Hamlet. Testing programs with the aid of a compiler. IEEE TSE, pages 279-290, 1977.

Digital Library

[16]

M. Harman, Y. Jia, and W. B. Langdon. Strong higher order mutation-based test data generation. In Proc. FSE, pages 212-222, 2011.

Digital Library

[17]

M. J. Harrold, J. A. Jones, T. Li, D. Liang, A. Orso, M. Pennings, S. Sinha, S. A. Spoon, and A. Gujarathi. Regression test selection for java software. In Proc. OOPSLA, pages 312-326, 2001.

Digital Library

[18]

R. Hierons and M. Harman. Using program slicing to assist in the detection of equivalent mutants. STVR, 9(4):233-262, 1999.

[19]

W. E. Howden. Weak mutation testing and completeness of test sets. IEEE TSE, pages 371-379, 1982.

Digital Library

[20]

Y. Jia and M. Harman. An analysis and survey of the development of mutation testing. IEEE TSE, 37(5):649-678, 2011.

Digital Library

[21]

K. N. King and A. J. Offutt. A Fortran language system for mutation-based software testing. Software: Practice and Experience, 21(7):685-718, 1991.

Digital Library

[22]

A. Kinneer, M. Dwyer, and G. Rothermel. Sofya: A flexible framework for development of dynamic program analyses for java software. Technical report, CSE, UNL, 2006.

[23]

E. W. Krauser, A. P. Mathur, and V. Rego. High performance software testing on SIMD machines. IEEE TSE, 17(5):403-423, 1991.

Digital Library

[24]

Y. S. Ma, J. Offutt, and Y. R. Kwon. MuJava: An automated class mutation system. STVR, 15(2):97-133, 2005.

Digital Library

[25]

A. P. Mathur. Performance, effectiveness, and reliability issues in software testing. In Proc. COMPSAC, pages 604-605, 1991.

[26]

A. P. Mathur and E.W. Krauser. Mutant unification for improved vectorization. Technical report, Purdue University, 1988.

[27]

D. Melski and T. Reps. Interconvertibility of a class of set constraints and context-free-language reachability. Theoretical Computer Science, 248(1-2):29-98, 2000.

Digital Library

[28]

A. S. Namin, J. H. Andrews, and D. J. Murdoch. Sufficient mutation operators for measuring test effectiveness. In Proc. ICSE, pages 351-360, 2008.

Digital Library

[29]

A. J. Offutt and W. M. Craft. Using compiler optimization techniques to detect equivalent mutants. STVR, 4(3):131-154, 1994.

[30]

A. J. Offutt, Z. Jin, and J. Pan. The dynamic domain reduction approach to test data generation. Software Practice and Experience, 29(2):167-193, 1999.

Digital Library

[31]

A. J. Offutt, A. Lee, G. Rothermel, R. H. Untch, and C. Zapf. An experimental determination of sufficient mutation operators. ACM TOSEM, 5(2):99-118, 1996.

Digital Library

[32]

A. J. Offutt, J. Pan, and J. M. Voas. Procedures for reducing the size of coverage-based test sets. In Proc. International Conference on Testing Computer Software, 1995.

[33]

A. J. Offutt, R. P. Pargas, S. V. Fichter, and P. K. Khambekar. Mutation testing of software using MIMD computer. In Proc. International Conference on Parallel Processing, pages 257-266, 1992.

[34]

A. Orso, N. Shi, and M. J. Harrold. Scaling regression testing to large software systems. In Proc. FSE, pages 241-252, 2004.

Digital Library

[35]

S. Person, G. Yang, N. Rungta, and S. Khurshid. Directed incremental symbolic execution. In Proc. PLDI, pages 504-515, 2011.

Digital Library

[36]

T. Reps. Program analysis via graph reachability. Information and software technology, 40(11):701-726, 1998.

[37]

G. Rothermel and M. J. Harrold. A safe, efficient regression test selection technique. ACM TOSEM, 6(2):173-210, 1997.

Digital Library

[38]

D. Schuler, V. Dallmeier, and A. Zeller. Efficient mutation testing by checking invariant violations. In Proc. ISSTA, pages 69-80, 2009.

Digital Library

[39]

D. Schuler and A. Zeller. Javalanche: Efficient mutation testing for Java. In Proc. FSE, pages 297-298, 2009.

Digital Library

[40]

R. Untch, A. J. Offutt, and M. J. Harrold. Mutation analysis using program schemata. In Proc. ISSTA, pages 139-148, 1993.

Digital Library

[41]

W. E. Wong and A. P. Mathur. Reducing the cost of mutation testing: An empirical study. JSS, 31(3):185-196, 1995.

Digital Library

[42]

M. Woodward and K. Halewood. From weak to strong, dead or alive? an analysis of some mutation testing issues. In Proc. of the Second Workshop on Software Testing, Verification, and Analysis, pages 152-158, 1988.

[43]

Z. Xu, Y. Kim, M. Kim, G. Rothermel, and M. B. Cohen. Directed test suite augmentation: techniques and tradeoffs. In Proc. FSE, pages 257-266, 2010.

Digital Library

[44]

G. Yang, M. B. Dwyer, and G. Rothermel. Regression model checking. In Proc. ICSM, pages 115-124, 2009.

[45]

L. Zhang, S. S. Hou, J. J. Hu, T. Xie, and H. Mei. Is operator-based mutant selection superior to random mutant selection? In Proc. ICSE, pages 435-444, 2010.

Digital Library

[46]

L. Zhang, T. Xie, L. Zhang, N. Tillmann, J. de Halleux, and H. Mei. Test generation via dynamic symbolic execution for mutation testing. In Proc. ICSM, pages 1-10, 2010.

Digital Library

Cited By

Ojdanic MSoremekun EDegiovanni RPapadakis MLe Traon Y(2023)Mutation Testing in Evolving Systems: Studying the Relevance of Mutants to Code EvolutionACM Transactions on Software Engineering and Methodology10.1145/353078632:1(1-39)Online publication date: 13-Feb-2023
https://dl.acm.org/doi/10.1145/3530786
Zheng WDeng PGui KWu X(2023)An Abstract Syntax Tree based static fuzzing mutation for vulnerability evolution analysisInformation and Software Technology10.1016/j.infsof.2023.107194158:COnline publication date: 1-Jun-2023
https://dl.acm.org/doi/10.1016/j.infsof.2023.107194
Barboni MMorichetta APolini ACasoni F(2023)ReSuMo: a regression strategy and tool for mutation testing of solidity smart contractsSoftware Quality Journal10.1007/s11219-023-09637-132:1(225-253)Online publication date: 17-Jun-2023
https://doi.org/10.1007/s11219-023-09637-1
Show More Cited By

Index Terms

Regression mutation testing
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Prioritizing mutants to guide mutation testing
ICSE '22: Proceedings of the 44th International Conference on Software Engineering

Mutation testing offers concrete test goals (mutants) and a rigorous test efficacy criterion, but it is expensive due to vast numbers of mutants, many of which are neither useful nor actionable. Prior work has focused on selecting representative and ...
Predictive mutation testing
ISSTA 2016: Proceedings of the 25th International Symposium on Software Testing and Analysis

Mutation testing is a powerful methodology for evaluating test suite quality. In mutation testing, a large number of mutants are generated and executed against the test suite to check the ratio of killed mutants. Therefore, mutation testing is widely ...
Faster mutation testing inspired by test prioritization and reduction
ISSTA 2013: Proceedings of the 2013 International Symposium on Software Testing and Analysis

Mutation testing is a well-known but costly approach for determining test adequacy. The central idea behind the approach is to generate mutants, which are small syntactic transformations of the program under test, and then to measure for a given test ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ISSTA 2012: Proceedings of the 2012 International Symposium on Software Testing and Analysis

July 2012

341 pages

ISBN:9781450314541

DOI:10.1145/2338965

General Chair:
Mats Heimdahl,
Program Chair:
Zhendong Su

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

SIGPLAN: ACM Special Interest Group on Programming Languages

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 July 2012

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Conference

ISSTA '12

Sponsor:

SIGSOFT

ISSTA '12: International Symposium on Software Testing and Analysis

July 15 - 20, 2012

MN, Minneapolis, USA

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Sponsor:
sigsoft

34th ACM SIGSOFT International Symposium on Software Testing and Analysis

June 25 - 28, 2025

Trondheim , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

62
Total Citations
View Citations
721
Total Downloads

Downloads (Last 12 months)16
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ojdanic MSoremekun EDegiovanni RPapadakis MLe Traon Y(2023)Mutation Testing in Evolving Systems: Studying the Relevance of Mutants to Code EvolutionACM Transactions on Software Engineering and Methodology10.1145/353078632:1(1-39)Online publication date: 13-Feb-2023
https://dl.acm.org/doi/10.1145/3530786
Zheng WDeng PGui KWu X(2023)An Abstract Syntax Tree based static fuzzing mutation for vulnerability evolution analysisInformation and Software Technology10.1016/j.infsof.2023.107194158:COnline publication date: 1-Jun-2023
https://dl.acm.org/doi/10.1016/j.infsof.2023.107194
Barboni MMorichetta APolini ACasoni F(2023)ReSuMo: a regression strategy and tool for mutation testing of solidity smart contractsSoftware Quality Journal10.1007/s11219-023-09637-132:1(225-253)Online publication date: 17-Jun-2023
https://doi.org/10.1007/s11219-023-09637-1
Tian ZChen JZhu QYang JZhang L(2022)Learning to Construct Better Mutation FaultsProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering10.1145/3551349.3556949(1-13)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3551349.3556949
Barboni MMorichetta APolini A(2022)Smart contract testingProceedings of the 5th International Workshop on Emerging Trends in Software Engineering for Blockchain10.1145/3528226.3528370(21-24)Online publication date: 19-May-2022
https://dl.acm.org/doi/10.1145/3528226.3528370
Kim JJeon JHong SYoo S(2022)Predictive Mutation Analysis via the Natural Language Channel in Source CodeACM Transactions on Software Engineering and Methodology10.1145/351041731:4(1-27)Online publication date: 12-Jul-2022
https://dl.acm.org/doi/10.1145/3510417
Wei AYi PLi ZXie TMarinov DLam WDwyer MDamian DZeller A(2022)Preempting flaky tests via non-idempotent-outcome testsProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510170(1730-1742)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510170
Zhang PLi YMa WYang YChen LLu HZhou YXu B(2022)CBUA: A Probabilistic, Predictive, and Practical Approach for Evaluating Test Suite EffectivenessIEEE Transactions on Software Engineering10.1109/TSE.2020.301036148:3(1067-1096)Online publication date: 1-Mar-2022
https://doi.org/10.1109/TSE.2020.3010361
Mondal SNasre R(2022) MahtabJournal of Systems and Software10.1016/j.jss.2019.110403158:COnline publication date: 21-Apr-2022
https://dl.acm.org/doi/10.1016/j.jss.2019.110403
Ojdanić MMa WLaurent TChekam TVentresque APapadakis M(2022)On the use of commit-relevant mutantsEmpirical Software Engineering10.1007/s10664-022-10138-127:5Online publication date: 1-Sep-2022
https://dl.acm.org/doi/10.1007/s10664-022-10138-1
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten