research-article

Learning combinatorial interaction test generation strategies using hyperheuristic search

Authors:

Justyna PetkeAuthors Info & Claims

ICSE '15: Proceedings of the 37th International Conference on Software Engineering - Volume 1

Pages 540 - 550

Published: 16 May 2015 Publication History

Abstract

The surge of search based software engineering research has been hampered by the need to develop customized search algorithms for different classes of the same problem. For instance, two decades of bespoke Combinatorial Interaction Testing (CIT) algorithm development, our exemplar problem, has left software engineers with a bewildering choice of CIT techniques, each specialized for a particular task. This paper proposes the use of a single hyperheuristic algorithm that learns search strategies across a broad range of problem instances, providing a single generalist approach. We have developed a Hyperheuristic algorithm for CIT, and report experiments that show that our algorithm competes with known best solutions across constrained and unconstrained problems: For all 26 real-world subjects, it equals or outperforms the best result previously reported in the literature. We also present evidence that our algorithm's strong generic performance results from its unsupervised learning. Hyperheuristic search is thus a promising way to relocate CIT design intelligence from human to machine.

References

[1]

M. Harman, "The current state and future of search based software engineering," in 2007 Future of Software Engineering (FOSE'07), 2007, pp. 342--357. {Online}. Available: http://dx.doi.org/10.1109/FOSE.2007.29

Digital Library

[2]

B. Garvin, M. Cohen, and M. Dwyer, "Evaluating improvements to a meta-heuristic search for constrained interaction testing," Empirical Software Engineering, vol. 16, no. 1, pp. 61--102, 2011. {Online}. Available: http://dx.doi.org/10.1007/s10664-010-9135-7

Digital Library

[3]

M. Harman, E. Burke, J. Clark, and X. Yao, "Dynamic adaptive search based software engineering," in International symposium on Empirical software engineering and measurement (ESEM'12), 2012, pp. 1--8.

Digital Library

[4]

E. K. Burke, M. Gendreau, M. Hyde, G. Kendall, G. Ochoa, E. Ozcan, and R. Qu, "Hyper-heuristics: A survey of the state of the art," Journal of the Operational Research Society, no. 64, pp. 1695--1724, 2013.

[5]

S. Ali, L. C. Briand, H. Hemmati, and R. K. Panesar-Walawege, "A systematic review of the application and empirical investigation of search-based test-case generation," IEEE Transactions on Software Engineering, pp. 742--762, 2010.

Digital Library

[6]

M. Harman, A. Mansouri, and Y. Zhang, "Search based software engineering: Trends, techniques and applications," ACM Computing Surveys, vol. 45, no. 1, p. Article 11, November 2012.

Digital Library

[7]

O. R "A survey on search-based software design," Computer Science Review, vol. 4, no. 4, pp. 203--249, 2010.

Digital Library

[8]

C. Henard, M. Papadakis, G. Perrouin, J. Klein, P. Heymans, and Y. L. Traon, "Bypassing the combinatorial explosion: Using similarity to generate and prioritize t-wise test configurations for software product lines," IEEE Trans. Software Eng., vol. 40, no. 7, pp. 650--670, 2014. {Online}. Available: http://doi.ieeecomputersociety.org/10.1109/TSE.2014.2327020

Digital Library

[9]

C. Nie and H. Leung, "A survey of combinatorial testing," ACM Computing Surveys, vol. 43, no. 2, pp. 1--29, February 2011.

Digital Library

[10]

M. B. Cohen, P. B. Gibbons, W. B. Mugridge, and C. J. Colbourn, "Constructing test suites for interaction testing," in The 25th International Conference on Software Engineering, ser. ICSE'03, Washington, DC, USA, 2003, pp. 38--48.

Digital Library

[11]

J. Stardom, Metaheuristics and the Search for Covering and Packing Arrays. Thesis (M.Sc.)--Simon Fraser University, 2001. {Online}. Available: http://books.google.co.uk/books?id=BIkmYAAACAAJ

[12]

G. Dueck, "New optimization heuristics - the great deluge algorithm and the record-to-record travel," Journal of Computational Physics, vol. 104, pp. 86--92, 1993.

Digital Library

[13]

K. Nurmela, "Upper bounds for covering arrays by tabu search," Discrete Applied Mathematics, vol. 138, no. 1--2, pp. 143--152, 2004.

Digital Library

[14]

R. C. Bryce and C. J. Colbourn, "One-test-at-a-time heuristic search for interaction test suites," in Genetic and Evolutionary Computation Conference, GECCO'07, 2007, pp. 1082--1089.

Digital Library

[15]

D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton, "The AETG system: an approach to testing based on combinatorial design," IEEE Transactions on Software Engineering, vol. 23, no. 7, pp. 437--444, 1997.

Digital Library

[16]

Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, and J. Lawrence, "IPOG-IPOG-D: efficient test generation for multi-way combinatorial testing," Software Testing Verification and Reliability, vol. 18, pp. 125--148, September 2008.

Digital Library

[17]

B. Hnich, S. Prestwich, E. Selensky, and B. Smith, "Constraint models for the covering test problem," Constraints, vol. 11, pp. 199--219, 2006.

Digital Library

[18]

Y. Lei and K. Tai, "In-parameter-order: a test generation strategy for pairwise testing," in The 3rd International High-Assurance Systems Engineering Symposium, 1998, pp. 254--261.

Digital Library

[19]

A. Calvagna and A. Gargantini, "A formal logic approach to constrained combinatorial testing," Journal of Automated Reasoning, vol. 45, pp. 331--358, December 2010.

Digital Library

[20]

J. Martinez-Pena, J. Torres-Jimenez, N. Rangel-Valdez, and H. Avila-George, "A heuristic approach for constructing ternary covering arrays using trinomial coefficients," in The 12th Ibero-American conference on Advances in artificial intelligence, ser. IBERAMIA'10, 2010, pp. 572--581.

Digital Library

[21]

I. Segall, R. Tzoref-Brill, and E. Farchi, "Using binary decision diagrams for combinatorial test design," in Proceedings of the 2011 International Symposium on Software Testing and Analysis (ISSTA'11), 2011, pp. 254--264. {Online}. Available: http://doi.acm.org/10.1145/2001420.2001451

Digital Library

[22]

C. J. Colbourn, "Covering array tables," Available at http://www.public.asu.edu/~ccolbou/src/tabby/catable.html, 2012.

[23]

A. Calvagna, A. Gargantini, and P. Vavassori, "Combinatorial interaction testing with CITLAB," in The Sixth International Conference on Software Testing, Verification and Validation (ICST'13), 2013, pp. 376--382.

Digital Library

[24]

J. Torres-Jimenez and E. Rodriguez-Tello, "New bounds for binary covering arrays using simulated annealing," Information Sciences, vol. 185, no. 1, pp. 137--152, 2012.

Digital Library

[25]

B. J. Garvin, M. B. Cohen, and M. B. Dwyer, "An improved meta-heuristic search for constrained interaction testing," in Proceedings of the 2009 1st International Symposium on Search Based Software Engineering (SSBSE'09), 2009, pp. 13--22. {Online}. Available: http://dx.doi.org/10.1109/SSBSE.2009.25

Digital Library

[26]

R. E. Lopez-Herrejon, J. Ferrer, J. F. Chicano, E. N. Haslinger, A. Egyed, and E. Alba, "Towards a benchmark and a comparison framework for combinatorial interaction testing of software product lines," The Computing Research Repository (CoRR), vol. abs/1401.5367, 2014.

[27]

C. Le Goues, M. Dewey-Vogt, S. Forrest, and W. Weimer, "A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each," in The 34th International Conference on Software Engineering, ICSE'12, 2012, pp. 3--13.

Digital Library

[28]

J. Petke, S. Yoo, M. B. Cohen, and M. Harman, "Efficiency and early fault detection with lower and higher strength combinatorial interaction testing," in The 9th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE'13), August 2013, pp. 26--36.

Digital Library

[29]

J. Czerwonka, "Pairwise testing in real world," in Pacific Northwest Software Quality Conference, October 2006, pp. 419--430.

[30]

M. B. Cohen, M. B. Dwyer, and J. Shi, "Interaction testing of highly-configurable systems in the presence of constraints," in Proceedings of the 2007 international symposium on Software testing and analysis, ser. ISSTA'07. New York, NY, USA: ACM, 2007, pp. 129--139. {Online}. Available: http://doi.acm.org/10.1145/1273463.1273482

Digital Library

[31]

R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. MIT Press, 1998. {Online}. Available: http://www.cs.ualberta.ca/%7Esutton/book/ebook/the-book.html

Digital Library

[32]

Y.-W. Tung and W. Aldiwan, "Automating test case generation for the new generation mission software system," in IEEE Aerospace Conference 2000, vol. 1, 2000, pp. 431--437.

[33]

M. Chateauneuf and D. L. Kreher, "On the state of strength-three covering arrays," Journal of Combinatorial Designs, vol. 10, no. 4, pp. 217--238, 2002. {Online}. Available: http://dx.doi.org/10.1002/jcd.10002

[34]

M. B. Cohen, C. J. Colbourn, and A. C. H. Ling, "Augmenting simulated annealing to build interaction test suites," in The 14th International Symposium on Software Reliability Engineering, ser. ISSRE'03, Washington, DC, USA, 2003, pp. 394--405. {Online}. Available: http://dl.acm.org/citation.cfm?id=951952.952381

Digital Library

[35]

M. Cohen, M. Dwyer, and J. Shi, "Constructing interaction test suites for highly-configurable systems in the presence of constraints: A greedy approach," IEEE Transactions on Software Engineering, vol. 34, no. 5, pp. 633--650, 2008.

Digital Library

[36]

D. Richard Kuhn and V. Okum, "Pseudo-exhaustive testing for software," in The 30th Annual IEEE/NASA Software Engineering Workshop (SEW'06), 2006, pp. 153--158. {Online}. Available: http://dx.doi.org/10.1109/SEW.2006.26

Digital Library

[37]

H. Do, S. G. Elbaum, and G. Rothermel, "Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact". Empirical Software Engineering, vol. 10, no. 4, pp. 405--435, 2005.

Digital Library

[38]

P. Nayeri, "Post-optimization: Necessity analysis for combinatorial arrays," Ph.D. Thesis, Department of Computer Science and Engineering, Arizona State University, April 2011.

Digital Library

[39]

A. Calvagna and A. Gargantini, "T-wise combinatorial interaction test suites construction based on coverage inheritance," Software Testing, Verification and Reliability, vol. 22, no. 7, pp. 507--526, 2012. {Online}. Available: http://dx.doi.org/10.1002/stvr.466

Digital Library

[40]

Amazon, "EC2 (Elastic Compute Cloud)," Available at http://aws.amazon.com/ec2/.

Cited By

Luo CLyu SZhao QWu WZhang HHu CChristakis MPradel M(2024)Beyond Pairwise Testing: Advancing 3-wise Combinatorial Interaction Testing for Highly Configurable SystemsProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680309(641-653)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680309
Zhao QLuo CCai SWu WLin JZhang HHu CChandra SBlincoe KTonella P(2023)CAmpactor: A Novel and Effective Local Search Algorithm for Optimizing Pairwise Covering ArraysProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616284(81-93)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3616284
Luo CSong JZhao QLi YCai SHu C(2023)Generating Pairwise Covering Arrays for Highly Configurable Software SystemsProceedings of the 27th ACM International Systems and Software Product Line Conference - Volume A10.1145/3579027.3608998(261-267)Online publication date: 28-Aug-2023
https://dl.acm.org/doi/10.1145/3579027.3608998
Show More Cited By

Index Terms

Learning combinatorial interaction test generation strategies using hyperheuristic search

Recommendations

A Tabu-Search Hyperheuristic for Timetabling and Rostering

Hyperheuristics can be defined to be heuristics which choose between heuristics in order to solve a given optimisation problem. The main motivation behind the development of such approaches is the goal of developing automated scheduling methods which ...
A discrete gravitational search algorithm for solving combinatorial optimization problems

Metaheuristics are general search strategies that, at the exploitation stage, intensively exploit areas of the solution space with high quality solutions and, at the exploration stage, move to unexplored areas of the solution space when necessary. The ...
Evolutionary and population-based methods versus constructive search strategies in dynamic combinatorial optimization

A GRASP-based constructive algorithm is proposed to solve combinatorial dynamic optimization problems.Tests are conducted on time-varying multi-dimensional knapsack problem with dimensional changes.Improved evolutionary algorithms, employing hyper-...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '15: Proceedings of the 37th International Conference on Software Engineering - Volume 1

May 2015

999 pages

ISBN:9781479919345

General Chair:
Antonia Bertolino
ISTI-CNR, Italy
,
Program Chairs:
Gerardo Canfora
University of Sannio, Italy
,
Sebastian Elbaum
University of Nebraska-Lincoln

Sponsors

ACM: Association for Computing Machinery
SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS\DATC: IEEE Computer Society
TCSE: IEEE Computer Society's Tech. Council on Software Engin.

Publisher

IEEE Press

Publication History

Published: 16 May 2015

Check for updates

Qualifiers

Research-article

Conference

ICSE '15

Sponsor:

ACM
SIGSOFT
IEEE-CS\DATC
TCSE

ICSE '15: 37th International Conference on Software Engineering

May 16 - 24, 2015

Florence, Italy

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

27
Total Citations
View Citations
246
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)0

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Luo CLyu SZhao QWu WZhang HHu CChristakis MPradel M(2024)Beyond Pairwise Testing: Advancing 3-wise Combinatorial Interaction Testing for Highly Configurable SystemsProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680309(641-653)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680309
Zhao QLuo CCai SWu WLin JZhang HHu CChandra SBlincoe KTonella P(2023)CAmpactor: A Novel and Effective Local Search Algorithm for Optimizing Pairwise Covering ArraysProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616284(81-93)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3616284
Luo CSong JZhao QLi YCai SHu C(2023)Generating Pairwise Covering Arrays for Highly Configurable Software SystemsProceedings of the 27th ACM International Systems and Software Product Line Conference - Volume A10.1145/3579027.3608998(261-267)Online publication date: 28-Aug-2023
https://dl.acm.org/doi/10.1145/3579027.3608998
Luo CZhao QCai SZhang HHu CRoychoudhury ACadar CKim M(2022)SamplingCA: effective and efficient sampling-based pairwise testing for highly configurable software systemsProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3549155(1185-1197)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3549155
Cha SLee MLee SOh HDwyer MDamian DZeller A(2022)SymTunerProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510185(2068-2079)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510185
Luo CSun BQiao BChen JZhang HLin JLin QZhang DSpinellis DGousios GChechik MDi Penta M(2021)LS-sampling: an effective local search based sampling approach for achieving high t-wise coverageProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3468264.3468622(1081-1092)Online publication date: 20-Aug-2021
https://dl.acm.org/doi/10.1145/3468264.3468622
Luo CLin JCai SChen XHe BQiao BZhao PLin QZhang HWu WRajmohan SZhang D(2021)AutoCCAGProceedings of the 43rd International Conference on Software Engineering10.1109/ICSE43902.2021.00030(201-212)Online publication date: 22-May-2021
https://dl.acm.org/doi/10.1109/ICSE43902.2021.00030
Lin JCai SHe BFu YLuo CLin QJuristo N(2021)FastCAProceedings of the 43rd International Conference on Software Engineering: Companion Proceedings10.1109/ICSE-Companion52605.2021.00040(77-80)Online publication date: 25-May-2021
https://dl.acm.org/doi/10.1109/ICSE-Companion52605.2021.00040
Bahomaid AAlsewari AZamli KAlhendawi KAl-Janabi A(2020)A Kidney Algorithm with Elitism for Combinatorial Testing ProblemProceedings of the 2020 the 7th International Conference on Automation and Logistics (ICAL)10.1145/3412953.3412970(6-11)Online publication date: 22-Jul-2020
https://dl.acm.org/doi/10.1145/3412953.3412970
Cha SOh HDumas MPfahl DApel SRusso A(2019)Concolic testing with adaptively changing search heuristicsProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3338906.3338964(235-245)Online publication date: 12-Aug-2019
https://dl.acm.org/doi/10.1145/3338906.3338964
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents