research-article

A study of equivalent and stubborn mutation operators using human analysis of equivalence

Authors:

Mark Harman, and

Yue JiaAuthors Info & Claims

ICSE 2014: Proceedings of the 36th International Conference on Software Engineering

May 2014

Pages 919 - 930

https://doi.org/10.1145/2568225.2568265

Published: 31 May 2014 Publication History

Abstract

Though mutation testing has been widely studied for more than thirty years, the prevalence and properties of equivalent mutants remain largely unknown. We report on the causes and prevalence of equivalent mutants and their relationship to stubborn mutants (those that remain undetected by a high quality test suite, yet are non-equivalent). Our results, based on manual analysis of 1,230 mutants from 18 programs, reveal a highly uneven distribution of equivalence and stubbornness. For example, the ABS class and half UOI class generate many equivalent and almost no stubborn mutants, while the LCR class generates many stubborn and few equivalent mutants. We conclude that previous test effectiveness studies based on fault seeding could be skewed, while developers of mutation testing tools should prioritise those operators that we found generate disproportionately many stubborn (and few equivalent) mutants.

References

[1]

A. T. Acree. On Mutation. Phd thesis, Georgia Institute of Technology, Atlanta, Georgia, 1980.

Digital Library

[2]

K. Adamopoulos, M. Harman, and R. M. Hierons. How to Overcome the Equivalent Mutant Problem and Achieve Tailored Selective Mutation Using Co-evolution. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO’04), volume 3103 of LNCS, pages 1338–1349, Seattle, Washington, USA, 26th-30th, June 2004. Springer.

[3]

A. V. Aho, R. Sethi, and J. D. Ullman. Compilers: Principles, techniques and tools. Addison Wesley, 1986.

Digital Library

[4]

N. Alshahwan and M. Harman. State aware test case regeneration for improving web application test suite coverage and fault detection. In International Symposium on Software Testing and Analysis (ISSTA 2012), pages 45–55, Minneapolis, MN, USA, 2012.

Digital Library

[5]

A. Arcuri. It does matter how you normalise the branch distance in search based software testing. In International Conference on Software testing (ICST 2010), pages 205–214, Paris, France, 2010. IEEE Computer Society.

Digital Library

[6]

D. Baldwin and F. G. Sayward. Heuristics for Determining Equivalence of Program Mutations. Research Report 276, Yale University, New Haven, Connecticut, 1979.

[7]

J. M. Bieman and L. M. Ott. Measuring functional cohesion. IEEE Transactions on Software Engineering, 20(8):644–657, Aug. 1994.

Digital Library

[8]

D. Binkley, N. Gold, and M. Harman. An empirical study of static program slice size. ACM Transactions on Software Engineering and Methodology, 16(2):1–32, 2007.

Digital Library

[9]

C. Bird and T. Zimmermann. Assessing the value of branches with what-if analysis. In W. Tracz, M. P. Robillard, and T. Bultan, editors, 20th ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE-20), SIGSOFT/FSE’12, Cary, NC, USA - November 11 - 16, 2012. ACM, 2012. Article 45(1-10).

Digital Library

[10]

L. Bottaci and E. S. Mresa. Eﬃciency of mutation operators and selective mutation strategies: An empirical study. Software Testing, Verification and Reliability, 9(4):205–232, Dec. 1999.

[11]

T. A. Budd. Mutation analysis: Ideas, examples, problems and prospects. In Proceedings of the Summer School on Computer Program Testing, pages 129–148, Sogesta, June 1981.

[12]

T. A. Budd and D. Angluin. Two Notions of Correctness and Their Relation to Testing. Acta Informatica, 18(1):31–45, March 1982.

Digital Library

[13]

C. Cadar, P. Godefroid, S. Khurshid, C. S. Păsăreanu, K. Sen, N. Tillmann, and W. Visser. Symbolic execution for software testing in practice: preliminary assessment. In 33rd International Conference on Software Engineering (ICSE’11), pages 1066–1071, New York, NY, USA, 2011. ACM.

Digital Library

[14]

D. Clark and R. M. Hierons. Squeeziness: An information theoretic measure for avoiding fault masking. Information Processing Letters, 112(8–9):335 – 340, 2012.

Digital Library

[15]

R. A. DeMillo and A. J. Offutt. Constraint-Based Automatic Test Data Generation. IEEE Transactions on Software Engineering, 17(9):900–910, September 1991.

Digital Library

[16]

H. Do, S. Elbaum, and G. Rothermel. Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact. Empirical Software Engineering, 10(4):405 – 435, Oct. 2005.

Digital Library

[17]

P. G. Frankl, S. N. Weiss, and C. Hu. All-uses vs mutation testing: An experimental comparison of effectiveness. Journal of Systems Software, 38:235–253, 1997.

Digital Library

[18]

G. Fraser and A. Arcuri. Evosuite: automatic test suite generation for object-oriented software. In 8th European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE ’11), pages 416–419. ACM, September 5th - 9th 2011.

Digital Library

[19]

G. Fraser and A. Zeller. Mutation-driven generation of unit tests and oracles. In Proceedings of the Nineteenth International Symposium on Software Testing and Analysis (ISSTA 2010), pages 147–158, Trento, Italy, July 12-16, 2010, 2010. ACM.

Digital Library

[20]

M. Gligoric, A. Groce, C. Zhang, R. Sharma, M. A. Alipour, and D. Marinov. Comparing non-adequate test suites using coverage criteria. In International Symposium on Software Testing and Analysis (ISSTA 2013), pages 302–313, Lugano, Switzerland, 2013. ACM.

Digital Library

[21]

P. Godefroid, N. Klarlund, and K. Sen. DART: directed automated random testing. In V. Sarkar and M. W. Hall, editors, Proceedings of the ACM SIGPLAN 2005 Conference on Programming Language Design and Implementation, Chicago, IL, USA, June 12-15, 2005, pages 213–223. ACM, 2005.

Digital Library

[22]

B. J. M. Grün, D. Schuler, and A. Zeller. The Impact of Equivalent Mutants. In Proceedings of the 4th International Workshop on Mutation Analysis (MUTATION’09), pages 192–199, Denver, Colorado, 1-4 April 2009. IEEE Computer Society.

Digital Library

[23]

M. Harman, R. M. Hierons, and S. Danicic. The Relationship Between Program Dependence and Mutation Analysis. In Proceedings of the 1st Workshop on Mutation Analysis (MUTATION’00), pages 5–13, San Jose, California, 6-7 October 2001.

Digital Library

[24]

M. Harman, Y. Jia, and B. Langdon. Strong higher order mutation-based test data generation. In 8th European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE ’11), pages 212–222, New York, NY, USA, September 5th - 9th 2011. ACM.

Digital Library

[25]

M. Harman, M. Okunlawon, B. Sivagurunathan, and S. Danicic. Slice-based measurement of coupling. In R. Harrison, editor, 19th ICSE, Workshop on Process Modelling and Empirical Studies of Software Evolution, Boston, Massachusetts, USA, May 1997.

[26]

R. M. Hierons, M. Harman, and S. Danicic. Using program slicing to assist in the detection of equivalent mutants. Software Testing, Verification and Reliability, 9(4):233–262, 1999.

[27]

International Standards Organisation (ISO). International standards organisation: Programming languages — C. International standard, ISO/IEC 9899: 1990 (E), Dec. 1990.

[28]

Y. Jia and M. Harman. Constructing subtle faults using higher order mutation testing. In 8th International Working Conference on Source Code Analysis and Manipulation (SCAM’08), pages 249–258, Beijing, China, 2008. IEEE Computer Society.

[29]

Y. Jia and M. Harman. An analysis and survey of the development of mutation testing. IEEE Transactions on Software Engineering, 37(5):649 – 678, September–October 2011.

Digital Library

[30]

R. Just, M. D. Ernst, and G. Fraser. Using state infection conditions to detect equivalent mutants and speed up mutation analysis. In Proceedings of the Dagstuhl Seminar 13021: Symbolic Methods in Testing, volume abs/1303.2784, 2013.

[31]

K. N. King and A. J. Offutt. A Fortran Language System for Mutation-Based Software Testing. Software:Practice and Experience, 21(7):685–718, October 1991.

Digital Library

[32]

M. Kintis, M. Papadakis, and N. Malevris. Isolating first order equivalent mutants via second order mutation. In Proceedings of the 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation, ICST ’12, pages 701–710, Washington, DC, USA, 2012. IEEE Computer Society.

Digital Library

[33]

K. Lakhotia, M. Harman, and H. Gross. AUSTIN: An open source tool for search based software testing of C programs. Journal of Information and Software Technology, 55(1):112–125, January 2013.

Digital Library

[34]

K. Lakhotia, P. McMinn, and M. Harman. Automated test data generation for coverage: Haven’t we solved this problem yet? In 4th Testing Academia and Industry Conference — Practice And Research Techniques (TAIC PART’09), pages 95–104, Windsor, UK, 4th–6th September 2009.

Digital Library

[35]

K. Lakhotia, P. McMinn, and M. Harman. An empirical investigation into branch coverage for C programs using CUTE and AUSTIN. Journal of Systems and Software, 83(12):2379–2391, 2010.

Digital Library

[36]

L. J. Morell. A Theory of Fault-Based Testing. IEEE Transactions on Software Engineering, 16(8):844–857, August 1990.

Digital Library

[37]

S. Mouchawrab, L. C. Briand, Y. Labiche, and M. Di Penta. Assessing, comparing, and combining state machine-based testing and structural testing: A series of experiments. IEEE Transactions on Software Engineering, 37(2):161–187, 2011.

Digital Library

[38]

V. P. Nelson. Fault-tolerant computing: Fundamental concepts. IEEE Computer, 23(7):19–25, 1990.

Digital Library

[39]

A. J. Offutt and W. M. Craft. Using Compiler Optimization Techniques to Detect Equivalent Mutants. Software Testing, Verification and Reliability, 4(3):131–154, September 1994.

[40]

A. J. Offutt, A. Lee, G. Rothermel, R. H. Untch, and C. Zapf. An experimental determination of suﬃcient mutant operators. ACM Transactions on Software Engineering Methodology, 5:99–118, 1996.

Digital Library

[41]

A. J. Offutt and J. Pan. Detecting Equivalent Mutants and the Feasible Path Problem. In Proceedings of the 1996 Annual Conference on Computer Assurance, pages 224–236, Gaithersburg, Maryland, June 1996. IEEE Computer Society Press.

[42]

A. J. Offutt and J. Pan. Automatically Detecting Equivalent Mutants and Infeasible Paths. Software Testing, Verification and Reliability, 7(3):165–192, September 1997.

[43]

A. J. Offutt, J. Pan, K. Tewary, and T. Zhang. An experimental evaluation of data flow and mutation testing. Software Practice and Experience, 26:165–176, 1996.

Digital Library

[44]

A. J. Offutt, J. Pan, and J. Voas. Procedures for reducing the size of coverage-based test sets. In Proceedings of the 12th International Conference on Testing Computer Software, pages 111–123, June 1995.

[45]

A. J. Offutt, G. Rothermel, and C. Zapf. An experimental evaluation of selective mutation. In 15th International Conference on Software Engineering (ICSE 1993), pages 100–107. IEEE Computer Society Press, Apr. 1993.

Digital Library

[46]

M. Papadakis and N. Malevris. Automatic mutation test case generation via dynamic symbolic execution. In Proceedings of the 21st International Symposium on Software Reliability Engineering (ISSRE’10), California, USA, November 2010.

Digital Library

[47]

M. Patrick, M. Oriol, and J. A. Clark. MESSI: Mutant evaluation by static semantic interpretation. In Proceedings of the 7th International Workshop on Mutation Analysis (MUTATION’12), Montreal, Canada, 17 April 2012. IEEE Computer Society.

Digital Library

[48]

D. Schuler, V. Dallmeier, and A. Zeller. Eﬃcient Mutation Testing by Checking Invariant Violations. Technique report, Saarland University, Saarbrucken, Telefon, 2009.

[49]

D. Schuler and A. Zeller. (Un-)Covering Equivalent Mutants. In Proceedings of the 3rd International Conference on Software Testing Verification and Validation (ICST’10), Paris, France, 6 April 2010. IEEE Computer Society.

Digital Library

[50]

D. Schuler and A. Zeller. Covering and uncovering equivalent mutants. Software Testing, Verification and Reliability, 23(5):353–374, 2013.

[51]

R. H. Untch, A. J. Offutt, and M. J. Harrold. Mutation analysis using mutant schemata. In T. Ostrand and E. Weyuker, editors, Proceedings of the 1993 International Symposium on Software Testing and Analysis (ISSTA), pages 139–148, 1993.

Digital Library

[52]

J. Voas and G. McGraw. Software Fault Injection: Inoculating Programs Against Errors. John Wiley & Sons, 1997.

Digital Library

[53]

M. R. Woodward and K. Halewood. From weak to strong, dead or alive? an analysis of some mutation testing issues. In Proceedings of the Second Workshop on Software Testing, Verification, and Analysis, Banff, Canada, July 1988.

[54]

Y. Zhan and J. A. Clark. Search-based mutation testing for simulink models. In H.-G. Beyer and U.-M. O’Reilly, editors, Genetic and Evolutionary Computation Conference (GECCO 05), pages 1061–1068, Washington DC, USA, June 2005.

Digital Library

Cited By

Wei CYao XGong DLiu H(2024)Test Data Generation for Mutation Testing Based on Markov Chain Usage Model and Estimation of Distribution AlgorithmIEEE Transactions on Software Engineering10.1109/TSE.2024.335829750:3(551-573)Online publication date: Mar-2024
https://doi.org/10.1109/TSE.2024.3358297
AlBlwi SMarsit IKhaireddine BAyad ALoh JMili A(2024)Subsumption, correctness and relative correctness: Implications for software testingScience of Computer Programming10.1016/j.scico.2024.103177(103177)Online publication date: Jul-2024
https://doi.org/10.1016/j.scico.2024.103177
Wei CYao XGong DLiu HDang X(2024)Set evolution based test data generation for killing stubborn mutantsJournal of Systems and Software10.1016/j.jss.2024.112121216(112121)Online publication date: Oct-2024
https://doi.org/10.1016/j.jss.2024.112121
Show More Cited By

Index Terms

A study of equivalent and stubborn mutation operators using human analysis of equivalence
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

An intuitive approach to determine test adequacy in safety-critical software

Safety-critical software must adhere to stringent quality standards and is expected to be thoroughly tested. However, exhaustive testing of software is usually impractical. The two main challenges faced by a software testing team are generation of ...
Read More
Faster mutation testing inspired by test prioritization and reduction
ISSTA 2013: Proceedings of the 2013 International Symposium on Software Testing and Analysis

Mutation testing is a well-known but costly approach for determining test adequacy. The central idea behind the approach is to generate mutants, which are small syntactic transformations of the program under test, and then to measure for a given test ...
Read More
Study of trivial compiler equivalence on C++ object-oriented mutation operators
SAC '19: Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing

Trivial Compiler Equivalence (TCE) has been recently proposed as an effective technique to detect equivalences between programs, where two or more programs are equivalent if the compiler produces the same binary code. Mutation testing can greatly ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE 2014: Proceedings of the 36th International Conference on Software Engineering

May 2014

1139 pages

ISBN:9781450327565

DOI:10.1145/2568225

General Chair:
Pankaj Jalote
IIIT-Delhi, India
,
Program Chairs:
Lionel Briand
University of Luxembourg, Luxembourg
,
André van der Hoek
University of California, Irvine, USA

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

TCSE: IEEE Computer Society's Tech. Council on Software Engin.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 May 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '14

Sponsor:

SIGSOFT

ICSE '14: 36th International Conference on Software Engineering

May 31 - June 7, 2014

Hyderabad, India

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

114
Total Citations
View Citations
703
Total Downloads

Downloads (Last 12 months)48
Downloads (Last 6 weeks)7

Other Metrics

View Author Metrics

Citations

Cited By

Wei CYao XGong DLiu H(2024)Test Data Generation for Mutation Testing Based on Markov Chain Usage Model and Estimation of Distribution AlgorithmIEEE Transactions on Software Engineering10.1109/TSE.2024.335829750:3(551-573)Online publication date: Mar-2024
https://doi.org/10.1109/TSE.2024.3358297
AlBlwi SMarsit IKhaireddine BAyad ALoh JMili A(2024)Subsumption, correctness and relative correctness: Implications for software testingScience of Computer Programming10.1016/j.scico.2024.103177(103177)Online publication date: Jul-2024
https://doi.org/10.1016/j.scico.2024.103177
Wei CYao XGong DLiu HDang X(2024)Set evolution based test data generation for killing stubborn mutantsJournal of Systems and Software10.1016/j.jss.2024.112121216(112121)Online publication date: Oct-2024
https://doi.org/10.1016/j.jss.2024.112121
Felici RPozzi LFuria C(2024)HyperPUT: generating synthetic faulty programs to challenge bug-finding toolsEmpirical Software Engineering10.1007/s10664-023-10430-829:2Online publication date: 15-Jan-2024
https://dl.acm.org/doi/10.1007/s10664-023-10430-8
Görz PMathis BHassler KGüler EHolz TZeller AGopinath RCalandrino JTroncoso C(2023)Systematic assessment of fuzzers using mutation analysisProceedings of the 32nd USENIX Conference on Security Symposium10.5555/3620237.3620491(4535-4552)Online publication date: 9-Aug-2023
https://dl.acm.org/doi/10.5555/3620237.3620491
Ojdanic MPapadakis MHarman MChandra SBlincoe KTonella P(2023)Keeping Mutation Test Suites Consistent and Relevant with Long-Standing MutantsProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613089(2067-2071)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3613089
Du HPalepu VJones JJust RFraser G(2023)To Kill a Mutant: An Empirical Study of Mutation Testing KillsProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598090(715-726)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598090
Zhang WGuo SZhang HSui YXue YXu Y(2023)Challenging Machine Learning-Based Clone Detectors via Semantic-Preserving Code TransformationsIEEE Transactions on Software Engineering10.1109/TSE.2023.324011849:5(3052-3070)Online publication date: 1-May-2023
https://doi.org/10.1109/TSE.2023.3240118
Garg AOjdanic MDegiovanni RChekam TPapadakis MLe Traon Y(2023)Cerebro: Static Subsuming Mutant SelectionIEEE Transactions on Software Engineering10.1109/TSE.2022.314051049:1(24-43)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TSE.2022.3140510
Su RZhang ZZhou YYao Y(2023)Test Generation for Mutation Testing by Symbolic Execution2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security Companion (QRS-C)10.1109/QRS-C60940.2023.00098(55-61)Online publication date: 22-Oct-2023
https://doi.org/10.1109/QRS-C60940.2023.00098
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents