Article

Symbolic search-based testing

Authors:

Youssef Hassoun,

Kiran Lakhotia,

Tanja VosAuthors Info & Claims

ASE '11: Proceedings of the 26th IEEE/ACM International Conference on Automated Software Engineering

Pages 53 - 62

https://doi.org/10.1109/ASE.2011.6100119

Published: 06 November 2011 Publication History

Abstract

We present an algorithm for constructing fitness functions that improve the efficiency of search-based testing when trying to generate branch adequate test data. The algorithm combines symbolic information with dynamic analysis and has two key advantages: It does not require any change in the underlying test data generation technique and it avoids many problems traditionally associated with symbolic execution, in particular the presence of loops. We have evaluated the algorithm on industrial closed source and open source systems using both local and global search-based testing techniques, demonstrating that both are statistically significantly more efficient using our approach. The test for significance was done using a one-sided, paired Wilcoxon signed rank test. On average, the local search requires 23.41% and the global search 7.78% fewer fitness evaluations when using a symbolic execution based fitness function generated by the algorithm.

References

[1]

M. Harman, Y. Hassoun, K. Lakhotia, P. McMinn, and J. Wegener, "The Impact of Input Domain Reduction on Search-Based Test Data Generation," in ESEC/SIGSOFT FSE, 2007, pp. 93-101.

Digital Library

[2]

M. Harman and P. McMinn, "A theoretical and empirical study of search-based testing: Local, global, and hybrid search," IEEE TSE, vol. 36, no. 2, pp. 226-247, 2010.

Digital Library

[3]

B. Korel, "Automated software test data generation," IEEE TSE, vol. 16, no. 8, pp. 870-879, Aug. 1990.

Digital Library

[4]

K. Lakhotia, P. McMinn, and M. Harman, "An empirical investigation into branch coverage for C programs using CUTE and AUSTIN," The J. of Systems and Software, vol. 83, no. 12, pp. 2379-2391, Dec. 2010.

Digital Library

[5]

C. C. Michael, G. McGraw, and M. Schatz, "Generating software test data by evolution," IEEE TSE, vol. 27, no. 12, pp. 1085-1110, 2001.

Digital Library

[6]

R. P. Pargas, M. J. Harrold, and R. Peck, "Test-data generation using genetic algorithms," Soft. Test., Ver. and Rel., vol. 9, no. 4, pp. 263-282, 1999.

[7]

J. Wegener, A. Baresel, and H. Sthamer, "Evolutionary test environment for automatic structural testing," Information & Software Technology, vol. 43, no. 14, pp. 841-854, 2001.

[8]

M. Harman, A. Mansouri, and Y. Zhang, "Search based software engineering: A comprehensive analysis and review of trends techniques and applications," Department of Computer Science, King's College London, Tech. Rep., April 2009.

[9]

N. Tracey, J. A. Clark, K. Mander, and J. A. McDermid, "An automated framework for structural test-data generation," in ASE, 1998, pp. 285- 288.

Digital Library

[10]

A. Arcuri, "It does matter how you normalise the branch distance in search based software testing," in ICST, 2010, pp. 205-214.

Digital Library

[11]

P. Godefroid, N. Klarlund, and K. Sen, "DART: directed automated random testing," ACM SIGPLAN Notices, vol. 40, no. 6, pp. 213-223, Jun. 2005.

Digital Library

[12]

B. Beizer, Software Testing Techniques, 2nd edition. International Thomson Computer Press, 1990.

Digital Library

[13]

P. McMinn, "IGUANA: Input generation using automated novel algorithms. A plug and play research tool." Univ. Of Sheffield, Tech. Rep., 2007.

[14]

P. McMinn, M. Harman, Y. Hassoun, K. Lakhotia, and J. Wegener, "Input Domain Reduction through Irrelevant Variable Removal and its Effect on Local, Global and Hybrid Search-Based Structural Test Data Generation," IEEE TSE, To Appear (2011).

Digital Library

[15]

D. A. Wheeler, "More than a gigabuck: Estimating GNU/Linux's size," http://www.dwheeler.com/sloc/, Jun. 2001.

[16]

J. H. Holland, "Genetic algorithms and the optimal allocation of trials," SIAM J. of Computing, vol. 2, no. 2, pp. 88-105, Jun. 1973.

Digital Library

[17]

H. Pohlheim, "Evolutionary algorithms: Overview, methods and operators." documentation for: Genetic evolutionary algorithm toolbox for use with matlab version: toolbox 1.92 documentation 1.92," 1999.

[18]

D. Whitley, "The GENITOR algorithm and selection pressure: Why rank-based allocation of reproductive trials is best," Computer Science Dept., Colorado State University, Fort Collins, CO, Tech. Rep., 1989.

[19]

J. E. Baker, "Reducing bias and inefficiency in the selection algorithm," in Genetic Algorithms and their Applications (ICGA'87), J. J. Grefenstette, Ed., 1987, pp. 14-21.

Digital Library

[20]

H. Mühlenbein and D. Schlierkamp-Voosen, "Predictive models for the breeder genetic algorithm, I: Continuous parameter optimization," Evolutionary Computation, vol. 1, no. 1, pp. 25-49, 1993.

Digital Library

[21]

R Development Core Team, R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna, Austria, 2011, ISBN 3-900051-07-0. {Online}. Available: http://www.R-project.org.

[22]

M. Harman, K. Lakhotia, and P. McMinn, "A multi-objective approach to search-based test data generation," in GECCO, 2007, pp. 1098-1105.

Digital Library

[23]

K. Sen, D. Marinov, and G. Agha, "CUTE: a concolic unit testing engine for C," in ESEC/SIGSOFT FSE, 2005, pp. 263-272.

Digital Library

[24]

C. Cadar and D. R. Engler, "Execution generated test cases: How to make systems code crash itself," in Model Checking Software, 12th International SPIN Workshop, vol. 3639, 2005, pp. 2-23.

Digital Library

[25]

N. Tillmann and J. de Halleux, "Pex-white box test generation for.NET," in TAP, 2008, pp. 134-153.

Digital Library

[26]

K. Inkumsah and T. Xie, "Evacon: A framework for integrating evolutionary and concolic testing for object-oriented programs," in ASE, November 2007, pp. 425-428.

Digital Library

[27]

P. Tonella, "Evolutionary testing of classes," in ISSTA, 2004, pp. 119- 128.

Digital Library

[28]

K. Sen and G. Agha, "CUTE and jCUTE: Concolic unit testing and explicit path model-checking tools," in CAV, 2006, pp. 419-423.

Digital Library

[29]

K. Lakhotia, N. Tillman, M. Harman, and J. de Halleux, "FloPSy - Search-Based Floating Point Constraint Solving for Symbolic Execution," in ICTSS, 2010, pp. 142-157.

Digital Library

[30]

K. Lakhotia, M. Harman, and P. McMinn, "Handling dynamic data structures in search based testing." in GECCO, 2008, pp. 1759-1766.

Digital Library

Cited By

Xiao DGuo YLi YChen L(2024)Optimizing Search-Based Unit Test Generation with Large Language Models: An Empirical StudyProceedings of the 15th Asia-Pacific Symposium on Internetware10.1145/3671016.3674813(71-80)Online publication date: 24-Jul-2024
https://dl.acm.org/doi/10.1145/3671016.3674813
Bui TDo VLe MBui QLi XHandl J(2024)Dynamic Difficulty Coefficient in Search-Based Software Testing: Targeting to Hard Branch CoverageProceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3638530.3654411(711-714)Online publication date: 14-Jul-2024
https://dl.acm.org/doi/10.1145/3638530.3654411
Haltermann JJakobs MRichter CWehrheim H(2024)Parallel program analysis on path rangesScience of Computer Programming10.1016/j.scico.2024.103154238:COnline publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1016/j.scico.2024.103154
Show More Cited By

Recommendations

Automatically performing weak mutation with the aid of symbolic execution, concolic testing and search-based testing

Automating software testing activities can increase the quality and drastically decrease the cost of software development. Toward this direction, various automated test data generation tools have been developed. The majority of existing tools aim at ...
Search-based test case implantation for testing untested configurations
Abstract Context
Modern large-scale software systems are highly configurable, and thus require a large number of test cases to be implemented and revised for testing a variety of system configurations. This makes testing highly ...
Symbolic Input-Output Conformance Checking for Model-Based Mutation Testing

This paper presents an approach to use symbolic input output conformance checking for mutation-based test case generation. In this approach, a possibly non-deterministic action system model is used as basis for generating a number of mutants. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ASE '11: Proceedings of the 26th IEEE/ACM International Conference on Automated Software Engineering

November 2011

677 pages

ISBN:9781457716386

Sponsors

Publisher

IEEE Computer Society

United States

Publication History

Published: 06 November 2011

Check for updates

Qualifiers

Article

Conference

ASE '11

Sponsor:

ASE '11: Automated Software Engineering

November 6 - 10, 2011

Acceptance Rates

Overall Acceptance Rate 82 of 337 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

28
Total Citations
View Citations
156
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Xiao DGuo YLi YChen L(2024)Optimizing Search-Based Unit Test Generation with Large Language Models: An Empirical StudyProceedings of the 15th Asia-Pacific Symposium on Internetware10.1145/3671016.3674813(71-80)Online publication date: 24-Jul-2024
https://dl.acm.org/doi/10.1145/3671016.3674813
Bui TDo VLe MBui QLi XHandl J(2024)Dynamic Difficulty Coefficient in Search-Based Software Testing: Targeting to Hard Branch CoverageProceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3638530.3654411(711-714)Online publication date: 14-Jul-2024
https://dl.acm.org/doi/10.1145/3638530.3654411
Haltermann JJakobs MRichter CWehrheim H(2024)Parallel program analysis on path rangesScience of Computer Programming10.1016/j.scico.2024.103154238:COnline publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1016/j.scico.2024.103154
Wang SMao XCao ZGao YShen QPeng C(2023)NxtUnit: Automated Unit Test Generation for GoProceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering10.1145/3593434.3593443(176-179)Online publication date: 14-Jun-2023
https://dl.acm.org/doi/10.1145/3593434.3593443
Bui TBui QDo VSilva SPaquete L(2023)A Novel Fitness Function for Automated Software Test Case Generation Based on Nested Constraint HardnessProceedings of the Companion Conference on Genetic and Evolutionary Computation10.1145/3583133.3590727(791-794)Online publication date: 15-Jul-2023
https://dl.acm.org/doi/10.1145/3583133.3590727
Yan STao GLiu XZhai JMa SXu LZhang XDevanbu PCohen MZimmermann T(2020)Correlations between deep neural network model coverage criteria and model qualityProceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3368089.3409671(775-787)Online publication date: 8-Nov-2020
https://dl.acm.org/doi/10.1145/3368089.3409671
Zhu ZJiao LAuger AStützle T(2019)Improving search-based software testing by constraint-based genetic operatorsProceedings of the Genetic and Evolutionary Computation Conference10.1145/3321707.3321720(1435-1442)Online publication date: 13-Jul-2019
https://dl.acm.org/doi/10.1145/3321707.3321720
Xu XJiao LZhu ZTakadama K(2018)A dynamic fitness function for search based software testingProceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3205651.3205700(320-321)Online publication date: 6-Jul-2018
https://dl.acm.org/doi/10.1145/3205651.3205700
Fu ZSu Z(2017)Achieving high coverage for floating-point code via unconstrained programmingACM SIGPLAN Notices10.1145/3140587.306238352:6(306-319)Online publication date: 14-Jun-2017
https://dl.acm.org/doi/10.1145/3140587.3062383
Braione PDenaro GMattavelli APezzè MBultan TSen K(2017)Combining symbolic execution and search-based testing for programs with complex heap inputsProceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3092703.3092715(90-101)Online publication date: 10-Jul-2017
https://dl.acm.org/doi/10.1145/3092703.3092715
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten