research-article

Reducing the Cost of Mutation Testing with the Use of Primitive Arcs Concept

Authors:

Pedro Henrique Kuroishi,

Marcio Eduardo Delamaro,

José Carlos Maldonado,

Auri Marcelo Rizzo VincenziAuthors Info & Claims

SBQS '20: Proceedings of the XIX Brazilian Symposium on Software Quality

Article No.: 20, Pages 1 - 10

https://doi.org/10.1145/3439961.3439981

Published: 06 March 2021 Publication History

Abstract

Mutation testing is a testing criterion used to measure the quality of a test suite. In mutation, a test suite is executed against the set of mutants of a given program under testing. A score is computed to measure the adequacy of the test suite in detecting faults. Although powerful, mutation testing has two major drawbacks: the high-computational cost to generate and execute the set of generated mutants and the existence of equivalent mutants. In this paper, we present a preliminary experimental study to investigate the use of control-flow information, aiming to reduce the number of mutants. For this study, only a subset of mutants, defined by its location, is executed. Such location is determined by the set of primitive arcs of a given program under testing. Next, it is analyzed the relationship between minimal mutants and primitive arcs. Results indicate that the approach reduces the number of mutants and equivalent mutants and, in most cases, still maintains a high mutation score concerning full mutation. Moreover, the results also indicate that there is a concentration of minimal mutants on the nodes related to primitive arcs. Finally, we compare the effectiveness of our strategy over random mutant sampling.

References

[1]

Hiralal Agrawal. 1994. Dominators, Super Blocks, and Program Coverage. In Proceedings of the 21st ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (Portland, Oregon, USA) (POPL ’94). Association for Computing Machinery, New York, NY, USA, 25–34. https://doi.org/10.1145/174675.175935

Digital Library

[2]

Paul Ammann, Marcio Eduardo Delamaro, and Jeff Offutt. 2014. Establishing Theoretical Minimal Sets of Mutants. In Proceedings of the 2014 IEEE International Conference on Software Testing, Verification, and Validation(ICST ’14). IEEE Computer Society, Washington, DC, USA, 21–30. https://doi.org/10.1109/ICST.2014.13

Digital Library

[3]

Timothy A. Budd and Dana Angluin. 1982. Two Notions of Correctness and Their Relation to Testing. Acta Inf. 18, 1 (March 1982), 31–45. https://doi.org/10.1007/BF00625279

Digital Library

[4]

M. L. Chaim. 1991. POKE-TOOL – A Tool to Support Data Flow Based Structural Test of Programs. Master’s thesis. DCA/FEEC/UNICAMP, Campinas, SP. (in Portuguese).

[5]

T. Chusho. 1987. Test Data Selection and Quality Estimation Based on the Concept of Essential Branches for Path Testing. IEEE Transactions on Software Engineering SE-13, 5 (May 1987), 509–517. https://doi.org/10.1109/TSE.1987.233196

Digital Library

[6]

Marcio Delamaro, Marcos Lordello Chaim, and José Carlos Maldonado. 2018. Where Are the Minimal Mutants?. In Proceedings of the XXXII Brazilian Symposium on Software Engineering (Sao Carlos, Brazil) (SBES ’18). ACM, New York, NY, USA, 190–195. https://doi.org/10.1145/3266237.3266241

Digital Library

[7]

Marcio Eduardo Delamaro, Lin Deng, Vinicius Humberto Serapilha Durelli, Nan Li, and Jeff Offutt. 2014. Experimental Evaluation of SDL and One-Op Mutation for C. In Proceedings of the 2014 IEEE International Conference on Software Testing, Verification, and Validation(ICST ’14). IEEE Computer Society, Washington, DC, USA, 203–212. https://doi.org/10.1109/ICST.2014.33

Digital Library

[8]

M. E. Delamaro, L. Deng, N. Li, V. Durelli, and J. Offutt. 2014. Growing a Reduced Set of Mutation Operators. In 2014 Brazilian Symposium on Software Engineering. 81–90. https://doi.org/10.1109/SBES.2014.14

Digital Library

[9]

M. E. Delamaro, J. C. Maldonado, and A. M. R. Vincenzi. 2000. Proteum/IM 2.0: An Integrated Mutation Testing Environment. In Mutation 2000 Symposium. Kluwer Academic Publishers, San Jose, CA, 91–101.

[10]

Marcio Eduardo Delamaro, Jeff Offutt, and Paul Ammann. 2014. Designing Deletion Mutation Operators. In Proceedings of the 2014 IEEE International Conference on Software Testing, Verification, and Validation(ICST ’14). IEEE Computer Society, Washington, DC, USA, 11–20. https://doi.org/10.1109/ICST.2014.12

Digital Library

[11]

R. A. DeMillo, R. J. Lipton, and F. G. Sayward. 1978. Hints on Test Data Selection: Help for the Practicing Programmer. Computer 11, 4 (April 1978), 34–41. https://doi.org/10.1109/C-M.1978.218136

Digital Library

[12]

Vinicius H.S. Durelli, Marcio E. Delamaro, and Jeff Offutt. 2018. An experimental comparison of edge, edge-pair, and prime path criteria. Science of Computer Programming 152 (2018), 99 – 115. https://doi.org/10.1016/j.scico.2017.10.003

Digital Library

[13]

R. Gopinath, I. Ahmed, M. A. Alipour, C. Jensen, and A. Groce. 2017. Mutation Reduction Strategies Considered Harmful. IEEE Transactions on Reliability 66, 3 (2017), 854–874.

[14]

R. G. Hamlet. 1977. Testing Programs with the Aid of a Compiler. IEEE Trans. Softw. Eng. 3, 4 (July 1977), 279–290. https://doi.org/10.1109/TSE.1977.231145

Digital Library

[15]

D. Holling, S. Banescu, M. Probst, A. Petrovska, and A. Pretschner. 2016. Nequivack: Assessing Mutation Score Confidence. In 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW). 152–161. https://doi.org/10.1109/ICSTW.2016.29

[16]

Changbin Ji, Zhenyu Chen, Baowen Xu, and Zhihong Zhao. 2009. A Novel Method of Mutation Clustering Based on Domain Analysis. In Proceedings of the 21st International Conference on Software Engineering and Knowledge Engineering, SEKE 2009. 422–425.

[17]

Y. Jia and M. Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering 37, 5 (2011), 649–678.

Digital Library

[18]

René Just, Bob Kurtz, and Paul Ammann. 2017. Inferring Mutant Utility from Program Context. In Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis (Santa Barbara, CA, USA) (ISSTA 2017). ACM, New York, NY, USA, 284–294. https://doi.org/10.1145/3092703.3092732

Digital Library

[19]

J. C. Maldonado. 1991. Potential-Uses Criteria: A Contribution to the Structural Testing of Software. Ph.D. Dissertation. DCA/FEE/UNICAMP, Campinas, SP, Brazil. (in Portuguese).

[20]

Michaël Marcozzi, Sébastien Bardin, Nikolai Kosmatov, Mike Papadakis, Virgile Prevosto, and Loïc Correnson. 2018. Time to Clean Your Test Objectives. In Proceedings of the 40th International Conference on Software Engineering (Gothenburg, Sweden) (ICSE ’18). ACM, New York, NY, USA, 456–467. https://doi.org/10.1145/3180155.3180191

Digital Library

[21]

A. P. Mathur. 1991. Performance, effectiveness, and reliability issues in software testing. In [1991] Proceedings The Fifteenth Annual International Computer Software Applications Conference. 604–605. https://doi.org/10.1109/CMPSAC.1991.170248

[22]

T. J. McCabe. 1976. A Complexity Measure. IEEE Trans. Softw. Eng. 2, 4 (July 1976), 308–320. https://doi.org/10.1109/TSE.1976.233837

Digital Library

[23]

A. Jefferson Offutt and Jie Pan. 1997. Automatically Detecting Equivalent Mutants and Infeasible Paths. Softw. Test., Verif. Reliab. 7 (1997), 165–192.

[24]

A. J. Offutt, G. Rothermel, and C. Zapf. 1993. An experimental evaluation of selective mutation. In Proceedings of 1993 15th International Conference on Software Engineering. 100–107. https://doi.org/10.1109/ICSE.1993.346062

[25]

A. Jefferson Offutt and Ronald H. Untch. 2001. Mutation Testing for the New Century. Kluwer Academic Publishers, Norwell, MA, USA, Chapter Mutation 2000: Uniting the Orthogonal, 34–44. http://dl.acm.org/citation.cfm?id=571305.571314

[26]

Mike Papadakis, Christopher Henard, Mark Harman, Yue Jia, and Yves Le Traon. 2016. Threats to the Validity of Mutation-Based Test Assessment. In Proceedings of the 25th International Symposium on Software Testing and Analysis (Saarbrücken, Germany) (ISSTA 2016). Association for Computing Machinery, New York, NY, USA, 354–365. https://doi.org/10.1145/2931037.2931040

Digital Library

[27]

G. Petrovic and M. Ivankovic. 2017. State of Mutation Testing at Google. In 2018 IEEE/ACM 40th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP). 163–171.

[28]

Alessandro Viola Pizzoleto, Fabiano Cutigi Ferrari, Jeff Offutt, Leo Fernandes, and Márcio Ribeiro. 2019. A systematic literature review of techniques and metrics to reduce the cost of mutation testing. Journal of Systems and Software 157 (2019), 110388. https://doi.org/10.1016/j.jss.2019.07.100

[29]

D. Schuler and A. Zeller. 2010. (Un-)Covering Equivalent Mutants. In 2010 Third International Conference on Software Testing, Verification and Validation. 45–54. https://doi.org/10.1109/ICST.2010.30

Digital Library

[30]

Chang-ai Sun, Feifei Xue, Huai Liu, and Xiangyu Zhang. 2017. A Path-aware Approach to Mutant Reduction in Mutation Testing. Inf. Softw. Technol. 81, C (Jan. 2017), 65–81. https://doi.org/10.1016/j.infsof.2016.02.006

Digital Library

[31]

W. Eric Wong and Aditya P. Mathur. 1995. Reducing the Cost of Mutation Testing: An Empirical Study. J. Syst. Softw. 31, 3 (Dec. 1995), 185–196. https://doi.org/10.1016/0164-1212(94)00098-0

Digital Library

Cited By

Brasil ADe Souza MDomenicali FMachado RJunior J(2024)Mutation Testing as a Quality Assurance Technique in a Fintech Company: An Experience Report in The IndustryProceedings of the XXIII Brazilian Symposium on Software Quality10.1145/3701625.3701629(446-451)Online publication date: 5-Nov-2024
https://dl.acm.org/doi/10.1145/3701625.3701629
Yik LNasir bin Wan Kadir WIbrahim N(2023)A Systematic Literature Review on Solutions of Mutation Testing Problems2023 IEEE 8th International Conference On Software Engineering and Computer Systems (ICSECS)10.1109/ICSECS58457.2023.10256324(64-71)Online publication date: 25-Aug-2023
https://doi.org/10.1109/ICSECS58457.2023.10256324

Recommendations

Prioritizing mutants to guide mutation testing
ICSE '22: Proceedings of the 44th International Conference on Software Engineering

Mutation testing offers concrete test goals (mutants) and a rigorous test efficacy criterion, but it is expensive due to vast numbers of mutants, many of which are neither useful nor actionable. Prior work has focused on selecting representative and ...
Mutation testing cost reduction by clustering overlapped mutants

We defined the term conditionally-overlapped (c-overlapped) mutants.C-overlapped mutants are expected to produce the same results against a test case.Clustering c-overlapped mutants effectively reduces the cost of mutation testing.Clustering c-...
Faster mutation testing inspired by test prioritization and reduction
ISSTA 2013: Proceedings of the 2013 International Symposium on Software Testing and Analysis

Mutation testing is a well-known but costly approach for determining test adequacy. The central idea behind the approach is to generate mutants, which are small syntactic transformations of the program under test, and then to measure for a given test ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

SBQS '20: Proceedings of the XIX Brazilian Symposium on Software Quality

December 2020

430 pages

ISBN:9781450389235

DOI:10.1145/3439961

Editors:
Ivan Machado
(UFBA)
,
Heitor Costa
(UFLA)
,
Adriano Albuquerque
(UNIFOR)
,
Suzana Sampaio
(UFRPE)
,
Rodrigo Santos
(UNIRIO)
,
Sheila Reinehr
(PUCPR)
,
Gleison Santos
(UNIRIO)
,
Sandro Bezerra
(UFPA)
,
Davi Viana
(UFMA)
,
Adriano Bessa Albuquerque
(UNIFOR)
,
Ana Regina Cavalcanti da Rocha
(COPPE/UFRJ)
,
Gleison Santos
(UNIRIO)
,
Ivan Machado
(UFBA)
,
Marcello Thiry
(UNIVALI)
,
Monalessa Perini Barcellos
(UFES)
,
Sheila Reinehr
(PUCPR)
,
Tayana Conte
(UFAM)
,
Davi Viana
(UFMA)
,
Marcelo Schots
(UERJ)

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 March 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

SBQS'20

SBQS'20: 19th Brazilian Symposium on Software Quality

December 1 - 4, 2020

São Luís, Brazil

Acceptance Rates

Overall Acceptance Rate 35 of 99 submissions, 35%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
69
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)0

Reflects downloads up to 25 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Brasil ADe Souza MDomenicali FMachado RJunior J(2024)Mutation Testing as a Quality Assurance Technique in a Fintech Company: An Experience Report in The IndustryProceedings of the XXIII Brazilian Symposium on Software Quality10.1145/3701625.3701629(446-451)Online publication date: 5-Nov-2024
https://dl.acm.org/doi/10.1145/3701625.3701629
Yik LNasir bin Wan Kadir WIbrahim N(2023)A Systematic Literature Review on Solutions of Mutation Testing Problems2023 IEEE 8th International Conference On Software Engineering and Computer Systems (ICSECS)10.1109/ICSECS58457.2023.10256324(64-71)Online publication date: 25-Aug-2023
https://doi.org/10.1109/ICSECS58457.2023.10256324

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten