research-article

Directed test suite augmentation: techniques and tradeoffs

Authors:

Gregg Rothermel,

Myra B. CohenAuthors Info & Claims

FSE '10: Proceedings of the eighteenth ACM SIGSOFT international symposium on Foundations of software engineering

Pages 257 - 266

https://doi.org/10.1145/1882291.1882330

Published: 07 November 2010 Publication History

Abstract

Test suite augmentation techniques are used in regression testing to identify code elements affected by changes and to generate test cases to cover those elements. Our preliminary work suggests that several factors influence the cost and effectiveness of test suite augmentation techniques. These include the order in which affected elements are considered while generating test cases, the manner in which existing regression test cases and newly generated test cases are used, and the algorithm used to generate test cases. In this work, we present the results of an empirical study examining these factors, considering two test case generation algorithms (concolic and genetic). The results of our experiment show that the primary factor affecting augmentation is the test case generation algorithm utilized; this affects both cost and effectiveness. The manner in which existing and newly generated test cases are utilized also has a substantial effect on efficiency but a lesser effect on effectiveness. The order in which affected elements are considered turns out to have relatively few effects when using concolic test case generation, but more substantial effects when using genetic test case generation.

References

[1]

A. Aho, R. Sethi, and J. Ullman. Compilers, Principles, Techniques, and Tools. Addison-Wesley, Boston, MA, 1986.

Digital Library

[2]

T. Apiwattanapong, R. Santelices, P. K. Chittimalli, A. Orso, and M. J. Harrold. Matrix: Maintenance-oriented testing requirements identifier and examiner. In Test.: Acad. Ind. Conf. Pract. Res. Techn., pages 137--146, Aug. 2006.

Digital Library

[3]

S. Artzi, A. Kiezun, J. Dolby, F. Tip, D. Dig, A. Paradkar, and M. D. Ernst. Finding bugs in dynamic web applications. In Proc. Int'l Symp. Softw. Test. and Anal., July 2008.

Digital Library

[4]

A. Baresel, D. Binkley, M. Harman, and B. Korel. Evolutionary testing in the presence of loop-assigned flags: a testability transformation approach. In Proc. Int'l. Symp. Softw. Test. Anal., pages 108--118, July 2004.

Digital Library

[5]

D. Binkley. Semantics guided regression test cost reduction. IEEE Trans. Softw. Eng., 23(8), Aug. 1997.

Digital Library

[6]

S. Bohner and R. Arnold. Software Change Impact Analysis. IEEE Computer Society Press, Los Alamitos, CA, 1996.

Digital Library

[7]

C. Cadar, V. Ganesh, P. M. Pawlowski, D. L. Dill, and D. R. Engler. Exe: Automatically generating inputs of death. In Proc. Conf. Comp. Comm. Sec., pages 322--335, Oct 2006.

Digital Library

[8]

T. Y. Chen and R. Merkel. Quasi-random testing. IEEE Trans. Rel., 56(3):562--568, 2007.

[9]

L. Clarke. A system to generate test data and symbolically execute programs. IEEE Trans. Softw. Eng., 2(3):215--222, Sept. 1976.

Digital Library

[10]

L. Clarke and D. Richardson. Applications of symbolic evaluation. J. Sys. Softw., 5(1):15--35, Jan. 1985.

Digital Library

[11]

CREST - automatic test generation tool for C. http://code.google.com/p/crest/.

[12]

R. DeMillo and A. Offutt. Constraint-based automatic test data generation. IEEE TSE, 17(9):900--910, Sept. 1991.

Digital Library

[13]

E. Díaz, J. Tuya, R. Blanco, and J. Javier Dolado. A tabu search algorithm for structural software testing. Comp. Op. Res., 35(10):3052--3072, 2008.

Digital Library

[14]

H. Do, S. G. Elbaum, and G. Rothermel. Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact. Emp. Softw. Eng.: Int'l J., 10(4):405--435, 2005.

Digital Library

[15]

S. Elbaum, A. Malishevsky, and G. Rothermel. Test case prioritization: A family of empirical studies. IEEE Trans. Softw. Eng., 28(2):159--182, 2002.

Digital Library

[16]

M. Emmi, R. Majumdar, and K. Sen. Dynamic test input generation for database applications. In Proc. Int'l Symp. Softw. Test. Anal., pages 151--162, July 2007.

Digital Library

[17]

R. Ferguson and B. Korel. The chaining approach for software test data generation. ACM Trans. Softw. Eng. Meth., 5(1):63--86, Jan. 1996.

Digital Library

[18]

P. Godefroid, N. Klarlund, and K. Sen. DART: Directed automated random testing. In Proc. Conf. Prog. Lang. Des. Impl., pages 213--223, June 2005.

Digital Library

[19]

A. Gotlieb, B. Botella, and M. Reuher. Automatic test data generation using constraint solving techniques. In Proc. Int'l. Symp. Softw. Test. Anal., pages 53--62, Mar. 1998.

Digital Library

[20]

R. Gupta, M. Harrold, and M. Soffa. Program slicing-based regression testing techniques. J. Softw. Test., Verif., Rel., 6(2):83--111, June 1996.

[21]

M. Hutchins, H. Foster, T. Goradia, and T. Ostrand. Experiments on the effectiveness of dataflow- and controlflow-based test adequacy criteria. In Proc. Int'l. Conf. Softw. Eng., pages 191--200, May 1994.

Digital Library

[22]

B. Korel. Automated software test data generation. IEEE Trans. Softw. Eng., 16(8):870--897, Aug. 1990.

Digital Library

[23]

Z. Li, M. Harman, and R. Hierons. Search algorithms for regression test case prioritization. IEEE Trans. Softw. Eng., 33(4):225--237, Apr. 2007.

Digital Library

[24]

D. Marinov and S. Khurshid. TestEra: A novel framework for automated testing of Java programs. In Proc. Int'l. Conf. Auto. Softw. Eng., Nov. 2001.

Digital Library

[25]

P. McMinn. Search-based software test data generation: A survey. J. Softw. Test. Verif. Reliab., 14(2):105--156, 2004.

Digital Library

[26]

C. Michael, G. McGraw, and M. Shatz. Generating software test data by evolution. IEEE Trans. Softw. Eng., 27(12):1085--1110, Dec. 2001.

Digital Library

[27]

J. Offutt and A. Abdurazik. Generating tests from UML specifications. In Proc. Int'l. Conf. UML, Oct. 1999.

Digital Library

[28]

A. Orso, N. Shi, and M. J. Harrold. Scaling regression testing to large software systems. In Proc. Int'l. Symp. Found. Softw. Eng., Nov. 2004.

Digital Library

[29]

S. Person, M. B. Dwyer, S. Elbaum, and C. S. Păasăareanu. Differential symbolic execution. In Proc. Int'l. Symp. Found. Softw. Eng., pages 226--237, Nov. 2008.

Digital Library

[30]

G. Rothermel and M. J. Harrold. Selecting tests and identifying test coverage requirements for modified software. In Proc. Int'l Symp. Softw. Test. Anal., Aug 1994.

Digital Library

[31]

G. Rothermel and M. J. Harrold. A safe, efficient regression test selection technique. ACM Trans. Softw. Eng. Meth., 6(2):173--210, Apr. 1997.

Digital Library

[32]

R. Santelices, P. K. Chittimalli, T. Apiwattanapong, A. Orso, and M. J. Harrold. Test-suite augmentation for evolving software. In Proc. Int'l Conf. Auto. Softw. Eng., Sept. 2008.

Digital Library

[33]

K. Sen and G. Agha. JCUTE: Concolic unit testing and explicit path model-checking tools. In Proc. Int'l Conf. Comp. Aided Verif., pages 419--423, Aug 2006.

Digital Library

[34]

K. Sen, D. Marinov, and G. Agha. CUTE: A concolic unit testing engine for C. In Proc. Int'l Symp. Found. Softw. Eng., pages 263--272, Sept. 2005.

Digital Library

[35]

P. Tonella. Evolutionary testing of classes. In Intl. Symp. Softw. Test. Anal., pages 119--128, 2004.

Digital Library

[36]

W. Visser, C. Pasareanu, and S. Khurshid. Test input generation with Java Pathfinder. In Proc. Int'l Symp. Softw. Test. Anal., pages 97--107, July 2004.

Digital Library

[37]

H. Waeselynck, P. Thévenod-Fosse, and O. Abdellatif-Kaddour. Simulated annealing applied to test generation: Landscape characterization and stopping criteria. Emp. Softw. Eng., 12(1):35--63, 2007.

Digital Library

[38]

A. Walcott, M. L. Soffa, G. M. Kapfhammer, and R. S. Roos. Time-aware test suite prioritization. In Proc. Int'l. Conf. Softw. Test. Anal., pages 1--12, July 2006.

Digital Library

[39]

S. Wappler and F. Lammermann. Using evolutionary algorithms for the unit testing of object-oriented software. In Conf. Gen. Evol. Comp., pages 1053--1060, 2005.

Digital Library

[40]

G. Wassermann, D. Yu, A. Chander, D. Dhurjati, H. Inamura, and Z. Su. Dynamic test input generation for web applications. In Proc. Int'l Symp. Softw. Test. Anal., pages 249--260, July 2008.

Digital Library

[41]

Z. Xu, M. Cohen, and G. Rothermel. Factors affecting the use of genetic algorithms in test suite augmentation. In Gen. Evol. Comp. Conf, July 2010.

Digital Library

[42]

Z. Xu and G. Rothermel. Directed test suite augmentation. In Proc. Asia-Pacific Softw. Eng. Conf., Dec. 2009.

Digital Library

Cited By

Brandt CKhatami AWessel MZaidman A(2024)Shaken, Not Stirred: How Developers Like Their Amplified TestsIEEE Transactions on Software Engineering10.1109/TSE.2024.338101550:5(1264-1280)Online publication date: May-2024
https://doi.org/10.1109/TSE.2024.3381015
Wen TLi YZhang LMa HPan Z(2024)An Empirical Study on the Distance Metric in Guiding Directed Grey-box Fuzzing2024 IEEE 35th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE62328.2024.00038(307-318)Online publication date: 28-Oct-2024
https://doi.org/10.1109/ISSRE62328.2024.00038
Ghorbani NJabbarvand RSalehnamadi NGarcia JMalek S(2023)DeltaDroid: Dynamic Delivery Testing in AndroidACM Transactions on Software Engineering and Methodology10.1145/356321332:4(1-26)Online publication date: 26-May-2023
https://dl.acm.org/doi/10.1145/3563213
Show More Cited By

Index Terms

Directed test suite augmentation: techniques and tradeoffs
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Directed test suite augmentation
ICSE '11: Proceedings of the 33rd International Conference on Software Engineering

Test suite augmentation techniques are used in regression testing to identify code elements affected by changes and to generate test cases to cover those elements. Whereas methods and techniques to find affected elements have been extensively researched ...
Factors affecting the use of genetic algorithms in test suite augmentation
GECCO '10: Proceedings of the 12th annual conference on Genetic and evolutionary computation

Test suite augmentation techniques are used in regression testing to help engineers identify code elements affected by changes, and generate test cases to cover those elements. Researchers have created various approaches to identify affected code ...
Directed test suite augmentation: an empirical investigation

Test suite augmentation techniques are used in regression testing to identify code elements in a modified program that are not adequately tested and to generate test cases to cover those elements. A defining feature of test suite augmentation techniques ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

FSE '10: Proceedings of the eighteenth ACM SIGSOFT international symposium on Foundations of software engineering

November 2010

302 pages

ISBN:9781605587912

DOI:10.1145/1882291

General Chair:
Gruia-Catalin Roman
Washington University in St. Louis, USA
,
Program Chair:
André van der Hoek
University of California, Irvine, USA

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 November 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGSOFT/FSE'10

Sponsor:

SIGSOFT

SIGSOFT/FSE'10: 18th ACM SIGSOFT Symposium on the Foundations of Software Engineering

November 7 - 11, 2010

New Mexico, Santa Fe, USA

Acceptance Rates

Overall Acceptance Rate 17 of 128 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

74
Total Citations
View Citations
600
Total Downloads

Downloads (Last 12 months)40
Downloads (Last 6 weeks)4

Reflects downloads up to 16 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Brandt CKhatami AWessel MZaidman A(2024)Shaken, Not Stirred: How Developers Like Their Amplified TestsIEEE Transactions on Software Engineering10.1109/TSE.2024.338101550:5(1264-1280)Online publication date: May-2024
https://doi.org/10.1109/TSE.2024.3381015
Wen TLi YZhang LMa HPan Z(2024)An Empirical Study on the Distance Metric in Guiding Directed Grey-box Fuzzing2024 IEEE 35th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE62328.2024.00038(307-318)Online publication date: 28-Oct-2024
https://doi.org/10.1109/ISSRE62328.2024.00038
Ghorbani NJabbarvand RSalehnamadi NGarcia JMalek S(2023)DeltaDroid: Dynamic Delivery Testing in AndroidACM Transactions on Software Engineering and Methodology10.1145/356321332:4(1-26)Online publication date: 26-May-2023
https://dl.acm.org/doi/10.1145/3563213
Ojdanic MSoremekun EDegiovanni RPapadakis MLe Traon Y(2023)Mutation Testing in Evolving Systems: Studying the Relevance of Mutants to Code EvolutionACM Transactions on Software Engineering and Methodology10.1145/353078632:1(1-39)Online publication date: 13-Feb-2023
https://dl.acm.org/doi/10.1145/3530786
Brandt CWang DZaidman A(2023)When to Let the Developer Guide: Trade-offs Between Open and Guided Test Amplification2023 IEEE 23rd International Working Conference on Source Code Analysis and Manipulation (SCAM)10.1109/SCAM59687.2023.00032(231-241)Online publication date: 2-Oct-2023
https://doi.org/10.1109/SCAM59687.2023.00032
Bertolino ADe Angelis GDi Giandomenico FLonetti F(2023)Cross-coverage testing of functionally equivalent programs2023 IEEE/ACM International Conference on Automation of Software Test (AST)10.1109/AST58925.2023.00014(101-111)Online publication date: May-2023
https://doi.org/10.1109/AST58925.2023.00014
Wang PZhou XYue TLin PLiu YLu K(2023)The progress, challenges, and perspectives of directed greybox fuzzingSoftware Testing, Verification and Reliability10.1002/stvr.186934:2Online publication date: 14-Dec-2023
https://doi.org/10.1002/stvr.1869
Ojdanić MMa WLaurent TChekam TVentresque APapadakis M(2022)On the use of commit-relevant mutantsEmpirical Software Engineering10.1007/s10664-022-10138-127:5Online publication date: 1-Sep-2022
https://dl.acm.org/doi/10.1007/s10664-022-10138-1
Ma WChekam TPapadakis MHarman M(2021)MuDeltaProceedings of the 43rd International Conference on Software Engineering10.1109/ICSE43902.2021.00086(897-909)Online publication date: 22-May-2021
https://dl.acm.org/doi/10.1109/ICSE43902.2021.00086
Sykora KAhmed BBures M(2021)Code Coverage Aware Test Generation Using Constraint SolverSoftware Engineering and Formal Methods. SEFM 2020 Collocated Workshops10.1007/978-3-030-67220-1_5(58-66)Online publication date: 17-Jan-2021
https://doi.org/10.1007/978-3-030-67220-1_5
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents