research-article

Faster mutation analysis via equivalence modulo states

Authors:

Yangqingwei Shi,

Dan HaoAuthors Info & Claims

ISSTA 2017: Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis

Pages 295 - 306

https://doi.org/10.1145/3092703.3092714

Published: 10 July 2017 Publication History

Abstract

Mutation analysis has many applications, such as asserting the quality of test suites and localizing faults. One important bottleneck of mutation analysis is scalability. The latest work explores the possibility of reducing the redundant execution via split-stream execution. However, split-stream execution is only able to remove redundant execution before the first mutated statement.

In this paper we try to also reduce some of the redundant execution after the execution of the first mutated statement. We observe that, although many mutated statements are not equivalent, the execution result of those mutated statements may still be equivalent to the result of the original statement. In other words, the statements are equivalent modulo the current state. In this paper we propose a fast mutation analysis approach, AccMut. AccMut automatically detects the equivalence modulo states among a statement and its mutations, then groups the statements into equivalence classes modulo states, and uses only one process to represent each class. In this way, we can significantly reduce the number of split processes. Our experiments show that our approach can further accelerate mutation analysis on top of split-stream execution with a speedup of 2.56x on average.

References

[1]

Iftekhar Ahmed, Carlos Jensen, Alex Groce, and Paul E McKenney. 2017. Applying Mutation Analysis on Kernel Test Suites: An Experience Report. In ICSTW. 110–115.

[2]

J. H. Andrews, L. C. Briand, and Y. Labiche. 2005. Is mutation an appropriate tool for testing experiments?. In Proc. ICSE. 402–411.

Digital Library

[3]

Randal E Bryant, O’Hallaron David Richard, and O’Hallaron David Richard. 2003.

[4]

Computer systems: a programmer’s perspective. Prentice Hall Upper Saddle River.

[5]

Thierry Titcheu Chekam, Mike Papadakis, and Yves Le Traon. 2016. Assessing and Comparing Mutation-based Fault Localization Techniques. arXiv preprint arXiv:1607.05512 (2016).

[6]

Richard A DeMillo, Richard J Lipton, and Frederick G Sayward. 1978. Hints on test data selection: Help for the practicing programmer. Computer 11, 4 (1978), 34–41. Faster Mutation Analysis via Equivalence Modulo States ISSTA’17, July 10-14, 2017, Santa Barbara, CA, USA

Digital Library

[7]

Xavier Devroey, Gilles Perrouin, Maxime Cordy, Mike Papadakis, Axel Legay, and Pierre-Yves Schobbens. 2014. A Variability Perspective of Mutation Analysis. In FSE. 841–844.

Digital Library

[8]

Xavier Devroey, Gilles Perrouin, Mike Papadakis, Axel Legay, Pierre-Yves Schobbens, and Patrick Heymans. 2016. Featured model-based mutation analysis. In ICSE. 655–666.

Digital Library

[9]

Hyunsook Do, Sebastian G. Elbaum, and Gregg Rothermel. 2005. Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact. Empirical Software Engineering: An International Journal 10, 4 (2005), 405–435.

Digital Library

[10]

Vinicius HS Durelli, Jeff Offutt, and Marcio E Delamaro. 2012. Toward harnessing high-level language virtual machines for further speeding up weak mutation testing. In ICST. 681–690.

[11]

Gordon Fraser and Andrea Arcuri. 2015.

[12]

Achieving scalable mutation-based generation of whole test suites. Empirical Software Engineering 20, 3 (2015), 783–812.

Digital Library

[13]

Gordon Fraser and Andreas Zeller. 2012. Mutation-driven generation of unit tests and oracles. IEEE Transactions on Software Engineering 38, 2 (2012), 278–292.

Digital Library

[14]

Juan P Galeotti, Carlo A Furia, Eva May, Gordon Fraser, and Andreas Zeller. 2015.

[15]

Inferring loop invariants by mutation, dynamic analysis, and static checking. IEEE Transactions on Software Engineering 41, 10 (2015), 1019–1037.

[16]

Milos Gligoric, Vilas Jagannath, Qingzhou Luo, and Darko Marinov. 2013. Efficient mutation testing of multithreaded code. Software Testing, Verification and Reliability 23, 5 (2013), 375–403.

[17]

Milos Gligoric, Lingming Zhang, Cristiano Pereira, and Gilles Pokam. 2013.

[18]

Selective mutation testing for concurrent code. In Proc. ISSTA. 224–234.

[19]

Rahul Gopinath, Amin Alipour, Iftekhar Ahmed, Carlos Jensen, and Alex Groce. 2016. Measuring Effectiveness of Mutant Sets. In IEEE ICSTW. 132–141.

[20]

Rahul Gopinath, Mohammad Amin Alipour, Iftekhar Ahmed, Carlos Jensen, and Alex Groce. 2016. On the limits of mutation reduction strategies. In ICSE. 511–522.

Digital Library

[21]

Rahul Gopinath, Carlos Jensen, and Alex Groce. 2016. Topsy-Turvy: a smarter and faster parallelization of mutation analysis. In ICSE. 740–743.

Digital Library

[22]

Richard G Hamlet. 1977. Testing programs with the aid of a compiler. Software Engineering, IEEE Transactions on SE-3, 4 (1977), 279–290.

Digital Library

[23]

Mark Harman, Yue Jia, Pedro Reales Mateo, and Macario Polo. 2014.

[24]

Angels and monsters: An empirical investigation of potential test effectiveness and efficiency improvement from strongly subsuming higher order mutation. In ASE. 397–408.

[25]

Shin Hong, Byeongcheol Lee, Taehoon Kwak, Yiru Jeon, Bongsuk Ko, Yunho Kim, and Moonzoo Kim. 2015. Mutation-based fault localization for real-world multilingual programs (T). In ASE. 464–475.

[26]

Shan-Shan Hou, Lu Zhang, Tao Xie, Hong Mei, and Jia-Su Sun. 2007. Applying interface-contract mutation in regression testing of component-based software. In ICSM 2007. 174–183.

[27]

William E. Howden. 1982. Weak mutation testing and completeness of test sets. IEEE Transactions on Software Engineering SE-8, 4 (1982), 371–379.

Digital Library

[28]

Changbin Ji, Zhenyu Chen, Baowen Xu, and Zhihong Zhao. 2009.

[29]

A Novel Method of Mutation Clustering Based on Domain Analysis. In SEKE. 422–425.

[30]

Yue Jia and Mark Harman. 2011. An analysis and survey of the development of mutation testing. Software Engineering, IEEE Transactions on 37, 5 (2011), 649–678.

Digital Library

[31]

Ren é Just. 2014. The Major Mutation Framework: Efficient and Scalable Mutation Analysis for Java. In ISSTA. ACM, 433–436.

Digital Library

[32]

Ren é Just, Michael D Ernst, and Gordon Fraser. 2014. Efficient mutation analysis by propagating and partitioning infected execution states. In ISSTA. 315–326.

Digital Library

[33]

Ren é Just, Darioush Jalali, Laura Inozemtseva, Michael D Ernst, Reid Holmes, and Gordon Fraser. 2014. Are mutants a valid substitute for real faults in software testing?. In FSE. 654–665.

Digital Library

[34]

Ren é Just, Gregory M Kapfhammer, and Franz Schweiggert. 2012. Using nonredundant mutation operators and test suite prioritization to achieve efficient and scalable mutation analysis. In ISSRE. 11–20.

Digital Library

[35]

Ren é Just, Bob Kurtz, and Paul Ammann. 2017.

[36]

Customized Program Mutation: Inferring Mutant Utility from Program Context. Technical Report UM-CS-2017-004. University of Massachusetts, Amherst, Amherst, MA, USA.

[37]

Ren é Just and Franz Schweiggert. 2015. Higher accuracy and lower run time: Efficient mutation analysis using non-redundant mutation operators. Software Testing, Verification and Reliability (JSTVR) 25, 5-7 (2015), 490–507.

Digital Library

[38]

Rene Just, Franz Schweiggert, and Gregory M Kapfhammer. 2011. MAJOR: An efficient and extensible tool for mutation analysis in a Java compiler. In ASE. 612–615.

Digital Library

[39]

Dongsun Kim, Jaechang Nam, Jaewoo Song, and Sunghun Kim. 2013. Automatic patch generation learned from human-written patches. In ICSE ’13. 802–811.

Digital Library

[40]

K. N. King and A. Jefferson Offutt. 1991. A Fortran Language System for Mutationbased Software Testing. Softw. Pract. Exper. 21, 7 (June 1991), 685–718.

Digital Library

[41]

Edward W. Krauser, Aditya P. Mathur, and Vernon J Rego. 1991.

[42]

High performance software testing on SIMD machines. IEEE Transactions on Software Engineering 17, 5 (1991), 403–423.

Digital Library

[43]

Bob Kurtz, Paul Ammann, Jeff Offutt, M árcio E Delamaro, Mariet Kurtz, and Nida G ökc ¸e. 2016. Analyzing the validity of selective mutation with dominator mutants. In FSE. 571–582.

Digital Library

[44]

Chris Lattner and Vikram Adve. 2004.

[45]

LLVM: A compilation framework for lifelong program analysis & transformation. In CGO. 75–86.

[46]

Thomas Loise, Xavier Devroey, Gilles Perrouin, Mike Papadakis, and Patrick Heymans. 2017. Towards Security-aware Mutation Testing. In ICSTW. 97–102.

[47]

Yiling Lou, Dan Hao, and Lu Zhang. 2015. Mutation-based test-case prioritization in software evolution. In ISSRE. 46–57.

Digital Library

[48]

Evan Martin and Tao Xie. 2007. A fault model and mutation testing of access control policies. In WWW. 667–676.

Digital Library

[49]

Shabnam Mirshokraie, Ali Mesbah, and Karthik Pattabiraman. 2015.

[50]

Guided mutation testing for javascript web applications. IEEE Transactions on Software Engineering 41, 5 (2015), 429–444.

[51]

Seokhyeon Moon, Yunho Kim, Moonzoo Kim, and Shin Yoo. 2014.

[52]

Ask the mutants: Mutating faulty programs for fault localization. In ICST. 153–162.

Digital Library

[53]

Tejeddine Mouelhi, Yves Le Traon, and Benoit Baudry. 2007. Mutation analysis for security tests qualification. In TAICPART-MUTATION 2007. 233–242.

Digital Library

[54]

A Jefferson Offutt and W Michael Craft. 1994.

[55]

Using compiler optimization techniques to detect equivalent mutants. Software Testing, Verification and Reliability 4, 3 (1994), 131–154.

[56]

A Jefferson Offutt, Roy P Pargas, Scott V Fichter, and Prashant K Khambekar. 1992. Mutation testing of software using a MIMD computer. In Proc. ICPP.

[57]

Kai Pan, Xintao Wu, and Tao Xie. 2013. Automatic test generation for mutation testing on database applications. In Proceedings of the 8th International Workshop on Automation of Software Test. 111–117.

Digital Library

[58]

Mike Papadakis, Christopher Henard, Mark Harman, Yue Jia, and Yves Le Traon. 2016. Threats to the validity of mutation-based test assessment. In ISSTA. 354– 365.

Digital Library

[59]

Mike Papadakis, Christopher Henard, and Yves Le Traon. 2014. Sampling program inputs with mutation analysis: Going beyond combinatorial interaction testing. In ICST. 1–10.

Digital Library

[60]

Mike Papadakis, Yue Jia, Mark Harman, and Yves Le Traon. 2015. Trivial compiler equivalence: A large scale empirical study of a simple, fast and effective equivalent mutant detection technique. In ICSE. 936–946.

Digital Library

[61]

Mike Papadakis and Yves Le Traon. 2012. Using mutants to locate” unknown” faults. In ICST. 691–700.

Digital Library

[62]

Mike Papadakis and Yves Le Traon. 2014. Effective fault localization via mutation analysis: A selective mutation approach. In Proceedings of the 29th Annual ACM Symposium on Applied Computing. 1293–1300.

Digital Library

[63]

Mike Papadakis and Yves Le Traon. 2015. Metallaxis-FL: mutation-based fault localization. Software Testing, Verification and Reliability 25, 5-7 (2015), 605–628.

Digital Library

[64]

Spencer Pearson, Jos é Campos, Ren é Just, Gordon Fraser, Rui Abreu, Michael D. Ernst, Deric Pang, and Benjamin Keller. 2017.

[65]

Evaluating & improving fault localization techniques. In ICSE.

[66]

Yuhua Qi, Xiaoguang Mao, Yan Lei, Ziying Dai, and Chengsong Wang. 2014. The Strength of Random Search on Automated Program Repair. In Proceedings of the 36th International Conference on Software Engineering (ICSE 2014). 254–265.

Digital Library

[67]

David Schuler and Andreas Zeller. 2009. Javalanche: efficient mutation testing for Java. In ESEC/FSE. 297–298.

Digital Library

[68]

Francisco Carlos M Souza, Mike Papadakis, Yves Le Traon, and M árcio E Delamaro. 2016. Strong mutation-based test data generation using hill climbing. In IEEE/ACM SBST. 45–54.

Digital Library

[69]

Susumu Tokumoto, Hiroaki Yoshida, Kazunori Sakamoto, and Shinichi Honiden. 2016. MuVM: Higher Order Mutation Analysis Virtual Machine for C. In ICST. 320–329.

[70]

Roland H Untch, A Jefferson Offutt, and Mary Jean Harrold. 1993.

[71]

Mutation analysis using mutant schemata. In Proc. ISSTA. 139–148.

Digital Library

[72]

W. Weimer, Z.P. Fry, and S. Forrest. 2013. Leveraging program equivalence for adaptive program repair: Models and first results. In ASE. 356–366.

Digital Library

[73]

Westley Weimer, ThanhVu Nguyen, Claire Le Goues, and Stephanie Forrest. 2009.

[74]

Automatically finding patches using genetic programming. In ICSE ’09. 364–374.

[75]

W Eric Wong and Aditya P Mathur. 1995. Reducing the cost of mutation testing: An empirical study. Journal of Systems and Software 31, 3 (1995), 185–196.

Digital Library

[76]

Xiangjuan Yao, Mark Harman, and Yue Jia. 2014.

[77]

A study of equivalent and stubborn mutation operators using human analysis of equivalence. In ICSE. 919–930.

[78]

Jie Zhang, Ziyi Wang, Lingming Zhang, Dan Hao, Lei Zang, Shiyang Cheng, and Lu Zhang. 2016. Predictive mutation testing. In ISSTA. 342–353.

Digital Library

[79]

Jie Zhang, Muyao Zhu, Dan Hao, and Lu Zhang. 2014. An empirical study on the scalability of selective mutation testing. In ISSRE. 277–287.

Digital Library

[80]

Lu Zhang, Shan-Shan Hou, Jun-Jue Hu, Tao Xie, and Hong Mei. 2010. Is operatorbased mutant selection superior to random mutant selection?. In Proc. ICSE. 435–444.

Digital Library

[81]

Lingming Zhang, Darko Marinov, and Sarfraz Khurshid. 2013. Faster mutation testing inspired by test prioritization and reduction. In Proc. ISSTA. 235–245.

Digital Library

[82]

Lingming Zhang, Darko Marinov, Lu Zhang, and Sarfraz Khurshid. 2012. Regression mutation testing. In Proc. ISSTA. 331–341. ISSTA’17, July 10-14, 2017, Santa Barbara, CA, USA Bo Wang, Yingfei Xiong, Yangqingwei Shi, Lu Zhang, Dan Hao

Digital Library

[83]

Lingming Zhang, Tao Xie, Lu Zhang, Nikolai Tillmann, Jonathan De Halleux, and Hong Mei. 2010. Test generation via dynamic symbolic execution for mutation testing. In Proc. ICSM. 1–10.

Digital Library

Cited By

Xiao YYang CWang BXiong Y(2024)Accelerating Patch Validation for Program Repair With Interception-Based Execution SchedulingIEEE Transactions on Software Engineering10.1109/TSE.2024.335996950:3(618-635)Online publication date: Mar-2024
https://doi.org/10.1109/TSE.2024.3359969
Zhang QFang CMa YSun WChen Z(2023)A Survey of Learning-based Automated Program RepairACM Transactions on Software Engineering and Methodology10.1145/363197433:2(1-69)Online publication date: 23-Dec-2023
https://dl.acm.org/doi/10.1145/3631974
Garg AOjdanic MDegiovanni RChekam TPapadakis MLe Traon Y(2023)Cerebro: Static Subsuming Mutant SelectionIEEE Transactions on Software Engineering10.1109/TSE.2022.314051049:1(24-43)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TSE.2022.3140510
Show More Cited By

Index Terms

Faster mutation analysis via equivalence modulo states
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Predictive mutation testing
ISSTA 2016: Proceedings of the 25th International Symposium on Software Testing and Analysis

Mutation testing is a powerful methodology for evaluating test suite quality. In mutation testing, a large number of mutants are generated and executed against the test suite to check the ratio of killed mutants. Therefore, mutation testing is widely ...
Faster mutation analysis with fewer processes and smaller overheads
ASE '21: Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering

Mutation analysis is a powerful dynamic approach that has many applications, such as measuring the quality of test suites or automatically locating faults. However, the inherent low scalability hampers its practical use. To accelerate mutation analysis, ...
Efficient mutation analysis by propagating and partitioning infected execution states
ISSTA 2014: Proceedings of the 2014 International Symposium on Software Testing and Analysis

Mutation analysis evaluates a testing technique by measur- ing how well it detects seeded faults (mutants). Mutation analysis is hampered by inherent scalability problems — a test suite is executed for each of a large number of mutants. Despite ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ISSTA 2017: Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis

July 2017

447 pages

ISBN:9781450350761

DOI:10.1145/3092703

General Chair:
Tevfik Bultan
University of California at Santa Barbara, USA
,
Program Chair:
Koushik Sen
University of California at Berkeley, USA

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 July 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Distinguished Paper

Author Tags

Qualifiers

Research-article

Conference

ISSTA '17

Sponsor:

SIGSOFT

ISSTA '17: International Symposium on Software Testing and Analysis

July 10 - 14, 2017

CA, Santa Barbara, USA

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '24

Sponsor:
sigsoft

33rd ACM SIGSOFT International Symposium on Software Testing and Analysis

September 16 - 20, 2024

Vienna , Austria

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

39
Total Citations
View Citations
347
Total Downloads

Downloads (Last 12 months)36
Downloads (Last 6 weeks)3

Reflects downloads up to 10 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Xiao YYang CWang BXiong Y(2024)Accelerating Patch Validation for Program Repair With Interception-Based Execution SchedulingIEEE Transactions on Software Engineering10.1109/TSE.2024.335996950:3(618-635)Online publication date: Mar-2024
https://doi.org/10.1109/TSE.2024.3359969
Zhang QFang CMa YSun WChen Z(2023)A Survey of Learning-based Automated Program RepairACM Transactions on Software Engineering and Methodology10.1145/363197433:2(1-69)Online publication date: 23-Dec-2023
https://dl.acm.org/doi/10.1145/3631974
Garg AOjdanic MDegiovanni RChekam TPapadakis MLe Traon Y(2023)Cerebro: Static Subsuming Mutant SelectionIEEE Transactions on Software Engineering10.1109/TSE.2022.314051049:1(24-43)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TSE.2022.3140510
Viganò ECornejo OPastore FBriand L(2023)DaMAT: A Data-driven Mutation Analysis Tool2023 IEEE/ACM 45th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)10.1109/ICSE-Companion58688.2023.00047(165-169)Online publication date: May-2023
https://doi.org/10.1109/ICSE-Companion58688.2023.00047
Xiao YYang CWang BXiong Y(2023)ExpressAPR: Efficient Patch Validation for Java Automated Program Repair Systems2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)10.1109/ASE56229.2023.00012(2038-2041)Online publication date: 11-Sep-2023
https://doi.org/10.1109/ASE56229.2023.00012
Aşik SYayan U(2023)Generating Python Mutants From Bug Fixes Using Neural Machine TranslationIEEE Access10.1109/ACCESS.2023.330269511(85678-85693)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3302695
Vercammen SDemeyer SBorg MPettersson NHedin G(2023)Mutation testing optimisations using the Clang front‐endSoftware Testing, Verification and Reliability10.1002/stvr.1865Online publication date: 17-Oct-2023
https://doi.org/10.1002/stvr.1865
Cornejo OPastore FBriand LDwyer M(2022)MASSProceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings10.1145/3510454.3516840(134-138)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510454.3516840
Kim JJeon JHong SYoo S(2022)Predictive Mutation Analysis via the Natural Language Channel in Source CodeACM Transactions on Software Engineering and Methodology10.1145/351041731:4(1-27)Online publication date: 12-Jul-2022
https://dl.acm.org/doi/10.1145/3510417
Cornejo OPastore FBriand L(2022)Mutation Analysis for Cyber-Physical Systems: Scalable Solutions and Results in the Space DomainIEEE Transactions on Software Engineering10.1109/TSE.2021.310768048:10(3913-3939)Online publication date: 1-Oct-2022
https://doi.org/10.1109/TSE.2021.3107680
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents