Article

Combining search-based and constraint-based testing

Authors:

Gordon FraserAuthors Info & Claims

ASE '11: Proceedings of the 26th IEEE/ACM International Conference on Automated Software Engineering

Pages 436 - 439

https://doi.org/10.1109/ASE.2011.6100092

Published: 06 November 2011 Publication History

Abstract

Many modern automated test generators are based on either meta-heuristic search techniques or use constraint solvers. Both approaches have their advantages, but they also have specific drawbacks: Search-based methods get stuck in local optima and degrade when the search landscape offers no guidance; constraint-based approaches, on the other hand, can only handle certain domains efficiently. In this paper we describe a method that integrates both techniques and delivers the best of both worlds. On a high-level view, our method uses a genetic algorithm to generate tests, but the twist is that during evolution a constraint solver is used to ensure that mutated offspring efficiently explores different control flow. Experiments on 20 case study examples show that on average the combination improves branch coverage by 28% over search-based techniques and by 13% over constraint-based techniques.

References

[1]

M. Harman, L. Hu, R. Hierons, J. Wegener, H. Sthamer, A. Baresel, and M. Roper, "Testability transformation," IEEE Trans. Softw. Eng., vol. 30, pp. 3-16, January 2004.

Digital Library

[2]

K. Lakhotia, N. Tillmann, M. Harman, and J. de Halleux, "FloPSy - search-based floating point constraint solving for symbolic execution," in 22nd IFIP International Conference on Testing Software and Systems, ser. Lecture Notes in Computer Science. Springer Berlin / Heidelberg, 2010, pp. 142-157.

Digital Library

[3]

K. Inkumsah and T. Xie, "Improving structural testing of object-oriented programs via integrating evolutionary testing and symbolic execution," in Proceedings of the 2008 23rd IEEE/ACM International Conference on Automated Software Engineering (ASE'08). Washington, DC, USA: IEEE Computer Society, 2008, pp. 297-306.

Digital Library

[4]

W. Visser, K. Havelund, G. Brat, and S. Park, "Model checking programs," in Proceedings of the 15th IEEE International Conference on Automated Software Engineering, ser. ASE '00. Washington, DC, USA: IEEE Computer Society, 2000.

Digital Library

[5]

P. Godefroid, N. Klarlund, and K. Sen, "DART: directed automated random testing," in Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI'05). New York, NY, USA: ACM, 2005, pp. 213-223.

Digital Library

[6]

N. Tillmann and J. N. de Halleux, "Pex -- white box test generation for .NET," in International Conference on Tests And Proofs (TAP'08), ser. LNCS, vol. 4966. Springer, 2008, pp. 134-253.

Digital Library

[7]

P. McMinn, "Search-based software test data generation: a survey: Research articles," Software Testing Verification Reliability, vol. 14, no. 2, pp. 105-156, 2004.

Digital Library

[8]

G. Fraser and A. Arcuri, "Evolutionary generation of whole test suites," in Proceedings of the 11th International Conference On Quality Software (QSIC'11). IEEE Computer Society, 2011, pp. 31-40.

Digital Library

[9]

S. Anand, C. S. Pasareanu, and W. Visser, "JPF-SE: A symbolic execution extension to Java PathFinder," in Proceedings of the 13th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS'07). Berlin, Heidelberg: Springer-Verlag, 2007, pp. 134-138.

Digital Library

[10]

Choco Team, "Choco: an open source java constraint programming library," Ecole des Mines de Nantes, Research report 10-02-INFO, 2010. {Online}. Available: http://www.emn.fr/z-info/choco-solver/pdf/chocopresentation.pdf

[11]

R. Sharma, M. Gligoric, V. Jagannath, and D. Marinov, "A comparison of constraint-based and sequence-based generation of complex input data structures," in Proceedings of the 3rd International Conference on Software Testing, Verification, and Validation Workshops. Washington, DC, USA: IEEE Computer Society, 2010, pp. 337-342.

Digital Library

[12]

W. Visser, C. S. Pasareanu, and R. Pelánek, "Test input generation for Java containers using state matching," in Proceedings of the 2006 International Symposium on Software Testing and Analysis (ISSTA'06). New York, NY, USA: ACM, 2006, pp. 37-48.

Digital Library

[13]

M. Staats and C. Pasareanu, "Parallel symbolic execution for structural test generation," in Proceedings of the 19th International Symposium on Software Testing and Analysis (ISSTA'10). New York, NY, USA: ACM, 2010, pp. 183-194.

Digital Library

[14]

H. Sthamer, "The automatic generation of software test data using genetic algorithms," Ph.D. dissertation, University of Glamorgan, Pontyprid, Wales, UK, April 1996.

[15]

J. P. Galeotti, N. Rosner, C. G. López Pombo, and M. F. Frias, "Analysis of invariants for efficient bounded verification," in Proceedings of the 19th International Symposium on Software Testing and Analysis (ISSTA'10). New York, NY, USA: ACM, 2010, pp. 25-36.

Digital Library

[16]

E. Dorrer, "F-distribution," Commun. ACM, vol. 11, no. 2, pp. 116-117, 1968.

Digital Library

[17]

C. Schneckenburger and J. Mayer, "Towards the determination of typical failure patterns," in 4th International Workshop on Software Quality Assurance, co-located with ESEC/FSE'07 (SOQUA'07). New York, NY, USA: ACM, 2007, pp. 90-93.

Digital Library

[18]

H. Do, S. Elbaum, and G. Rothermel, "Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact," Empirical Software Engineering, vol. 10, pp. 405-435, October 2005.

Digital Library

[19]

R. Majumdar and K. Sen, "Hybrid concolic testing," in Proceedings of the 29th International Conference on Software Engineering (ICSE'07). Washington, DC, USA: IEEE Computer Society, 2007, pp. 416-426.

Digital Library

Cited By

Xiao DGuo YLi YChen L(2024)Optimizing Search-Based Unit Test Generation with Large Language Models: An Empirical StudyProceedings of the 15th Asia-Pacific Symposium on Internetware10.1145/3671016.3674813(71-80)Online publication date: 24-Jul-2024
https://dl.acm.org/doi/10.1145/3671016.3674813
Liu ZLiu KXia XYang XJust RFraser G(2023)Towards More Realistic Evaluation for Neural Test Oracle GenerationProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598080(589-600)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598080
Gereziher TGebrekrstos SGay GSilva SPaquete L(2023)Search-Based Test Generation Targeting Non-Functional Quality Attributes of Android AppsProceedings of the Genetic and Evolutionary Computation Conference10.1145/3583131.3590449(1518-1526)Online publication date: 15-Jul-2023
https://dl.acm.org/doi/10.1145/3583131.3590449
Show More Cited By

Recommendations

Search-based testing using constraint-based mutation

Many modern automated test generators are based on either metaheuristic search techniques or use constraint solvers. Both approaches have their advantages, but they also have specific drawbacks: Search-based methods may get stuck in local optima and ...
Boosting search based testing by using constraint based testing
SSBSE'12: Proceedings of the 4th international conference on Search Based Software Engineering

Search-Based Testing (SBT) uses an evolutionary algorithm to generate test cases. Traditionally, a random selection is used to generate an initial population and also, less often, during the evolution process. Such selection is likely to achieve lower ...
Intelligent search strategies based on adaptive Constraint Handling Rules

The most advanced implementation of adaptive constraint processing with Constraint Handling Rules (CHR) allows the application of intelligent search strategies to solve Constraint Satisfaction Problems (CSP). This presentation compares an improved ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ASE '11: Proceedings of the 26th IEEE/ACM International Conference on Automated Software Engineering

November 2011

677 pages

ISBN:9781457716386

Sponsors

Publisher

IEEE Computer Society

United States

Publication History

Published: 06 November 2011

Check for updates

Qualifiers

Article

Conference

ASE '11

Sponsor:

ASE '11: Automated Software Engineering

November 6 - 10, 2011

Acceptance Rates

Overall Acceptance Rate 82 of 337 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

26
Total Citations
View Citations
241
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 31 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Xiao DGuo YLi YChen L(2024)Optimizing Search-Based Unit Test Generation with Large Language Models: An Empirical StudyProceedings of the 15th Asia-Pacific Symposium on Internetware10.1145/3671016.3674813(71-80)Online publication date: 24-Jul-2024
https://dl.acm.org/doi/10.1145/3671016.3674813
Liu ZLiu KXia XYang XJust RFraser G(2023)Towards More Realistic Evaluation for Neural Test Oracle GenerationProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3597926.3598080(589-600)Online publication date: 12-Jul-2023
https://dl.acm.org/doi/10.1145/3597926.3598080
Gereziher TGebrekrstos SGay GSilva SPaquete L(2023)Search-Based Test Generation Targeting Non-Functional Quality Attributes of Android AppsProceedings of the Genetic and Evolutionary Computation Conference10.1145/3583131.3590449(1518-1526)Online publication date: 15-Jul-2023
https://dl.acm.org/doi/10.1145/3583131.3590449
Zhu ZJiao LAuger AStützle T(2019)Improving search-based software testing by constraint-based genetic operatorsProceedings of the Genetic and Evolutionary Computation Conference10.1145/3321707.3321720(1435-1442)Online publication date: 13-Jul-2019
https://dl.acm.org/doi/10.1145/3321707.3321720
Fellner AKrenn WSchlick RTarrach TWeissenbacher G(2019)Model-based, Mutation-driven Test-case Generation Via Heuristic-guided Branching SearchACM Transactions on Embedded Computing Systems10.1145/328925618:1(1-28)Online publication date: 25-Jan-2019
https://dl.acm.org/doi/10.1145/3289256
You WLiu XMa SPerry DZhang XLiang BAtlee JBultan TWhittle J(2019)SLFProceedings of the 41st International Conference on Software Engineering10.1109/ICSE.2019.00080(712-723)Online publication date: 25-May-2019
https://dl.acm.org/doi/10.1109/ICSE.2019.00080
Zhang XGong YWang Y(2017)Heuristic Guided Selective Path Exploration for Loop Structure in Coverage TestingInternational Journal of Open Source Software and Processes10.4018/IJOSSP.20170401048:2(59-75)Online publication date: 1-Apr-2017
https://dl.acm.org/doi/10.4018/IJOSSP.2017040104
Fellner AKrenn WSchlick RTarrach TWeissenbacher GTalpin JDerler PSchneider K(2017)Model-based, mutation-driven test case generation via heuristic-guided branching searchProceedings of the 15th ACM-IEEE International Conference on Formal Methods and Models for System Design10.1145/3127041.3127049(56-66)Online publication date: 29-Sep-2017
https://dl.acm.org/doi/10.1145/3127041.3127049
Braione PDenaro GMattavelli APezzè MBultan TSen K(2017)Combining symbolic execution and search-based testing for programs with complex heap inputsProceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3092703.3092715(90-101)Online publication date: 10-Jul-2017
https://dl.acm.org/doi/10.1145/3092703.3092715
Paduraru CMelemciuc MStefanescu A(2017)A distributed implementation using apache spark of a genetic algorithm applied to test data generationProceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3067695.3084219(1857-1863)Online publication date: 15-Jul-2017
https://dl.acm.org/doi/10.1145/3067695.3084219
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten