Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/2001420.2001451acmconferencesArticle/Chapter ViewAbstractPublication PagesisstaConference Proceedingsconference-collections
research-article

Using binary decision diagrams for combinatorial test design

Published: 17 July 2011 Publication History

Abstract

Combinatorial test design (CTD) is an effective test planning technique that reveals faulty feature interaction in a given system. The test space is modeled by a set of parameters, their respective values, and restrictions on the value combinations. A subset of the test space is then automatically constructed so that it covers all valid value combinations of every t parameters, where t is a user input. Various combinatorial testing tools exist, implementing different approaches to finding a set of tests that satisfies t-wise coverage. However, little consideration has been given to the process of defining the test space for CTD, which is usually a manual, labor-intensive, and error-prone effort. Potential errors include missing parameters and their values, wrong identification of parameters and of valid value combinations, and errors in the definition of restrictions that cause them not to capture the intended combinations. From our experience, lack of support for the test space definition process is one of the main obstacles in applying CTD to a wide range of testing domains.
In this work, we present a Cartesian product based methodology and technology that assist in defining a complete and consistent test space for CTD. We then show how using binary decision diagrams (BDDs) to represent and build the test space dramatically increases the scalability of our approach, making it applicable to large and complex real-life test design tasks, for which explicit representation of the test space is infeasible.
Finally, we show how BDDs can be used also to solve the CTD problem itself. We present a new and highly effective BDD-based approach for solving CTD, which finds a set of tests that satisfies t-wise coverage by subset selection. Our approach supports also advanced requirements such as requirements on the distribution of values in the selected tests. We apply our algorithm to real-life testing problems of varying complexity, and show its superior performance.

References

[1]
Allpairs website. http://www.mcdowella.demon.co.uk/allPairs.html.
[2]
B. Bollig and I. Wegener. Improving the Variable Ordering of OBDDs is NP-Complete. IEEE Transactions on Computers, 45:993--1002, Sept. 1996.
[3]
R. E. Bryant. Graph-Based Algorithms for Boolean Function Manipulation. IEEE Transactions on Computers, 35:677--691, 1986.
[4]
A. Calvagna and A. Gargantini. A Logic-Based Approach to Combinatorial Testing with Constraints. In Proc. 2nd International Conference on Tests and Proofs (TAP'08), pages 66--83. Springer-Verlag, 2008.
[5]
M. B. Cohen, M. B. Dwyer, and J. Shi. Interaction Testing of Highly-Configurable Systems in the Presence of Constraints. In Proc. 2007 International Symposium on Software Testing and Analysis (ISSTA'07), pages 129--139. ACM, 2007.
[6]
M. B. Cohen, P. B. Gibbons, W. B. Mugridge, and C. J. Colbourn. Constructing Test Suites for Interaction Testing. In Proc. 25th International Conference on Software Engineering (ICSE'03), pages 38--48. IEEE Computer Society, 2003.
[7]
C. J. Colbourn, M. B. Cohen, and R. Turban. A Deterministic Density Algorithm for Pairwise Interaction Coverage. In IASTED Conference on Software Engineering, pages 345--352. IASTED/ACTA Press, 2004.
[8]
CTD Benchmark Files. http://researcher.watson.ibm.com/researcher/files/il-ITAIS/ctdBenchmarks.zip.
[9]
J. Czerwonka. Pairwise Testing in Real World. In Proc. 24th Pacific Northwest Software Quality Conference (PNSQC'06), pages 419--430, 2006.
[10]
M. L. Fredman D. M. Cohen, S. R. Dalal and G. C. Patton. The AETG System: An Approach to Testing Based on Combinatorial Design. IEEE Transactions on Software Engineering, 23:437--444, 1997.
[11]
S. R. Dalal, A. Jain, N. Karunanithi, J. M. Leaton, C. M. Lott, G. C. Patton, and B. M. Horowitz. Model-Based Testing in Practice. In Proc. 21st International Conference on Software Engineering (ICSE'99), pages 285--294, 1999.
[12]
O. Grumberg E. M. Clarke and D. A. Peled. Model Checking. The MIT Press, 1999.
[13]
T. Erlz. SOA Principles of Service Design. Prentice Hall PTR, 2007.
[14]
FoCuS website. http://researcher.watson.ibm.com/-researcher/view_project.php?id=1871.
[15]
M. Grindal, J. Offutt, and S. F. Andler. Combination Testing Strategies: A Survey. Software Testing, Verification, and Reliability, 15:167--199, 2005.
[16]
A. Hartman. Software and Hardware Testing using Combinatorial Covering Suites. In Graph Theory, Combinatorics and Algorithms: Interdisciplinary Applications, pages 266--327. Springer-Verlag, 2005.
[17]
A. Hartman and L. Raskin. Problems and Algorithms for Covering Arrays. Discrete Mathematics, 284(1--3):149--156, 2004.
[18]
Jenny website. http://burtleburtle.net/bob/math/jenny.html.
[19]
D. R. Kuhn, D. R. Wallace, and A. M. Gallo. Software Fault Interactions and Implications for Software Testing. IEEE Transactions on Software Engineering, 30:418--421, 2004.
[20]
S. Kullback and R. A. Leibler. On Information and Sufficiency. Annals of Mathematical Statistics, 22(1):79--86, 1951.
[21]
S. Malik, A. R. Wang, R. K. Brayton, and A. Sangiovanni-Vincentelli. Logic Verification Using Binary Decision Diagrams in a Logic Synthesis Environment. In IEEE International Conference on Computer-Aided Design (ICCAD'88), pages 6--9. IEEE, 1988.
[22]
S. Minato. Graph-Based Representations of Discrete Functions. Representation of Discrete Functions, pages 1--27, 1996.
[23]
K. J. Nurmela. Upper Bounds for Covering Arrays by Tabu Search. Discrete Applied Mathematics, 138:143--152, 2004.
[24]
Pairwise testing website. http://www.pairwise.org.
[25]
Y. Lei R. Kuhn and R. Kacker. Practical Combinatorial Testing: Beyond Pairwise. IT Professional, 10:19--23, 2008.
[26]
Spec explorer. http://msdn.microsoft.com/en-us/-library/ee620448.aspx.
[27]
R. Brayton T. Kam, T. Villa and A. Sangiovanni-Vincentelli. Multi-Valued Decision Diagrams: Theory and Applications. Multiple-Valued Logic, 4(1--2):9--62, 1998.
[28]
K. C. Tai and Y. Lie. A Test Generation Strategy for Pairwise Testing. IEEE Transactions on Software Engineering, 28:109--111, 2002.
[29]
A. W. Williams and R. L Probert. A Measure for Component Interaction Test Coverage. In Proc. ACSI/IEEE International Conference on Computer Systems and Applications (AICCSA'01), pages 304--311, 2001.
[30]
Alan W. Williams. Determination of Test Configurations for Pair-Wise Interaction Coverage. In Proc. IFIP TC6/WG6.1 13th International Conference on Testing Communicating Systems: Tools and Techniques (TestCom'00), pages 59--74. Kluwer, B. V., 2000.

Cited By

View all
  • (2024)Use of binary decision diagrams to handle constraints in combinatorial interaction testing2024 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW60967.2024.00048(211-214)Online publication date: 27-May-2024
  • (2024)Effectively computing high strength mixed covering arrays with constraintsJournal of Parallel and Distributed Computing10.1016/j.jpdc.2023.104791185(104791)Online publication date: Mar-2024
  • (2023)A novel combinatorial testing approach with fuzzing strategyJournal of Software: Evolution and Process10.1002/smr.2537Online publication date: 2-Feb-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ISSTA '11: Proceedings of the 2011 International Symposium on Software Testing and Analysis
July 2011
394 pages
ISBN:9781450305624
DOI:10.1145/2001420
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 July 2011

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. BDDs
  2. CTD
  3. Cartesian products
  4. binary decision diagrams
  5. combinatorial test design
  6. test planning

Qualifiers

  • Research-article

Conference

ISSTA '11
Sponsor:

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)13
  • Downloads (Last 6 weeks)1
Reflects downloads up to 26 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Use of binary decision diagrams to handle constraints in combinatorial interaction testing2024 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW60967.2024.00048(211-214)Online publication date: 27-May-2024
  • (2024)Effectively computing high strength mixed covering arrays with constraintsJournal of Parallel and Distributed Computing10.1016/j.jpdc.2023.104791185(104791)Online publication date: Mar-2024
  • (2023)A novel combinatorial testing approach with fuzzing strategyJournal of Software: Evolution and Process10.1002/smr.2537Online publication date: 2-Feb-2023
  • (2022)Combinatorial Test Generation for Multiple Input Models With Shared ParametersIEEE Transactions on Software Engineering10.1109/TSE.2021.306595048:7(2606-2628)Online publication date: 1-Jul-2022
  • (2022)An Adaptive Penalty based Parallel Tabu Search for Constrained Covering Array GenerationInformation and Software Technology10.1016/j.infsof.2021.106768143:COnline publication date: 1-Mar-2022
  • (2022)Incomplete MaxSAT approaches for combinatorial testingJournal of Heuristics10.1007/s10732-022-09495-328:4(377-431)Online publication date: 1-Aug-2022
  • (2021)An environment for benchmarking combinatorial test suite generators2021 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW52544.2021.00021(48-56)Online publication date: Apr-2021
  • (2021)AutoCCAGProceedings of the 43rd International Conference on Software Engineering10.1109/ICSE43902.2021.00030(201-212)Online publication date: 22-May-2021
  • (2020)Constrained detecting arrays for fault localization in combinatorial testingProceedings of the 35th Annual ACM Symposium on Applied Computing10.1145/3341105.3373952(1971-1978)Online publication date: 30-Mar-2020
  • (2020)A Two-Step Heuristic Algorithm for Generating Constrained Detecting Arrays for Combinatorial Interaction Testing2020 IEEE 29th International Conference on Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE)10.1109/WETICE49692.2020.00050(219-224)Online publication date: Sep-2020
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media