research-article

Multi-thread Combinatorial Test Generation with SMT solvers

Authors:

Andrea Bombarda,

Angelo Gargantini,

Andrea CalvagnaAuthors Info & Claims

SAC '23: Proceedings of the 38th ACM/SIGAPP Symposium on Applied Computing

Pages 1698 - 1705

https://doi.org/10.1145/3555776.3577703

Published: 07 June 2023 Publication History

Abstract

Combinatorial interaction testing (CIT) is a testing technique that has proven to be effective in finding faults due to the interaction among inputs and in reducing the number of test cases, without losing effectiveness. Several tools have been proposed in the literature; however, generating tests remains a challenging task. In this paper, we present a technique for generating combinatorial test suites that uses a multi-thread architecture and exploits Satisfiability Modulo Theory (SMT) solvers to represent model parameters, constraints, and tuples, and it builds from SMT solver contexts the desired test suite. This technique is implemented by the tool KALI. The main advantage of using SMT solvers is that combinatorial models can contain all kinds of parameters and constraints. To evaluate our approach, we tested the impact of several optimizations and compared the performance of KALI with those of some existing tools for test generation. Our experiments confirm that the use of multi-threading is a promising technique but still requires some optimization for being more effective than the already available ones.

References

[1]

H. Avila George. 2012. Constructing Covering Arrays using Parallel Computing and Grid Computing. Ph. D. Dissertation. Universitat Politècnica de València.

[2]

A. Bombarda, E. Crippa, and A. Gargantini. 2021. An environment for benchmarking combinatorial test suite generators. In 2021 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE.

[3]

A. Bombarda and A. Gargantini. 2022. Parallel Test Generation for Combinatorial Models Based on Multivalued Decision Diagrams. In 2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE.

[4]

A. Bombarda, A. Gargantini, and A. Calvagna. 2022. Replication package for the paper "Multi-thread Combinatorial Test Generation with SMT solvers". https://github.com/fmselab/ct-tools/tree/main/KALI

[5]

A. Calvagna, A. Gargantini, and E. Tramontana. 2009. Building T-wise Combinatorial Interaction Test Suites by Means of Grid Computing. In 2009 18th IEEE International Workshops on Enabling Technologies: Infrastructures for Collaborative Enterprises. 213--218.

[6]

A. Calvagna, G. Pappalardo, and E. Tramontana. 2012. A Novel Approach to Effective Parallel Computing of t-Wise Covering Arrays. In 2012 IEEE 21st International Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprises. 149--153.

[7]

J. Christ, J. Hoenicke, and A. Nutz. 2012. SMTInterpol: An Interpolating SMT Solver. In Model Checking Software, Alastair Donaldson and David Parker (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 248--254.

[8]

A. Cimatti, A. Griggio, B. J. Schaafsma, and R. Sebastiani. 2013. The MathSAT5 SMT Solver. In Tools and Algorithms for the Construction and Analysis of Systems, Nir Piterman and Scott A. Smolka (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 93--107.

[9]

L. De Moura and N. Bjørner. 2008. Z3: An Efficient SMT Solver. In Tools and Algorithms for the Construction and Analysis of Systems, C. R. Ramakrishnan and Jakob Rehof (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 337--340.

[10]

B. Dutertre. 2014. Yices 2.2. In Computer-Aided Verification (CAV'2014) (Lecture Notes in Computer Science, Vol. 8559), Armin Biere and Roderick Bloem (Eds.). Springer, 737--744.

Digital Library

[11]

A. Gargantini and M. Radavelli. 2018. Migrating Combinatorial Interaction Test Modeling and Generation to the Web. In 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). 308--317.

[12]

W. Grieskamp et al. 2009. Interaction coverage meets path coverage by SMT constraint solving. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 5826 LNCS (2009), 97--112.

[13]

C. Henard, M. Papadakis, and Y. Le Traon. 2015. Flattening or not of the combinatorial interaction testing models?. In 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 1--4.

[14]

Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, and J. Lawrence. 2007. IPOG: A General Strategy for T-Way Software Testing. In 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems (ECBS'07). IEEE.

[15]

H. Mercan, C. Yilmaz, and K. Kaya. 2019. CHiP: A Configurable Hybrid Parallel Covering Array Constructor. IEEE Transactions on Software Engineering 45, 12 (2019), 1270--1291.

[16]

J. Peleska, E. Vorobev, and F. Lapschies. 2011. Automated Test Case Generation with SMT-Solving and Abstract Interpretation. In Proceedings of the Third International Conference on NASA Formal Methods (Pasadena, CA) (NFM'11). Springer-Verlag, Berlin, Heidelberg, 298--312.

[17]

P. Rümmer. 2008. A Constraint Sequent Calculus for First-Order Logic with Linear Integer Arithmetic. In Proc., 15th Int. Conf. on Logic for Programming, Artificial Intelligence and Reasoning (LNCS, Vol. 5330). Springer, 274--289.

Digital Library

[18]

K. C. Tai and Y. Lie. 2002. A Test Generation Strategy for Pairwise Testing. IEEE Trans. Softw. Eng. 28, 1 (jan 2002), 109--111.

[19]

M. Veanes, C. Campbell, W. Schulte, and N. Tillmann. 2005. Online Testing with Model Programs. SIGSOFT Softw. Eng. Notes 30, 5 (sep 2005), 273--282.

Digital Library

[20]

Y. Wang et al. 2022. An Adaptive Penalty based Parallel Tabu Search for Constrained Covering Array Generation. Information and Software Technology 143 (2022).

[21]

C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, and A. Wesslén. 2012. Experimentation in Software Engineering. Springer Berlin Heidelberg.

[22]

Mohammed I. Younis and Kamal Z. Zamli. 2010. MC-MIPOG: A Parallel t-Way Test Generation Strategy for Multicore Systems. ETRI Journal 32, 1 (feb 2010), 73--83.

[23]

L. Yu, F. Duan, Y. Lei, R. N. Kacker, and D. R. Kuhn. 2015. Constraint handling in combinatorial test generation using forbidden tuples. In 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW). 1--9.

[24]

L. Yu, Y. Lei, R. N. Kacker, and D. R. Kuhn. 2013. ACTS: A Combinatorial Test Generation Tool. In 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation. 370--375.

Cited By

Britikov KZlatkin IFedyukovich GAlt LSharygina N(2024)SolTG: A CHC-Based Solidity Test Case GeneratorComputer Aided Verification10.1007/978-3-031-65627-9_23(466-479)Online publication date: 26-Jul-2024
https://doi.org/10.1007/978-3-031-65627-9_23
Bombarda AGargantini A(2023)Incremental generation of combinatorial test suites starting from existing seed tests2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW58534.2023.00044(197-205)Online publication date: Apr-2023
https://doi.org/10.1109/ICSTW58534.2023.00044

Index Terms

Multi-thread Combinatorial Test Generation with SMT solvers
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Prioritizing random combinatorial test suites
SAC '17: Proceedings of the Symposium on Applied Computing

The behaviour of a system under test can be influenced by several factors, such as system configurations, user inputs, and so on. It has also been observed that many failures are caused by only a small number of factors. Combinatorial testing aims at ...
Evolutionary algorithm for prioritized pairwise test data generation
GECCO '12: Proceedings of the 14th annual conference on Genetic and evolutionary computation

Combinatorial Interaction Testing (CIT) is a technique used to discover faults caused by parameter interactions in highly configurable systems. These systems tend to be large and exhaustive testing is generally impractical. Indeed, when the resources ...
Generating combinatorial test suite for interaction relationship
SOQUA '07: Fourth international workshop on Software quality assurance: in conjunction with the 6th ESEC/FSE joint meeting

Combinatorial testing could detect the faults triggered by the interactions among factors in software. But in many cases, the pair-wise, N-way and even the variable strength combinatorial testing may lead test suite redundancy and fault detect ability ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SAC '23: Proceedings of the 38th ACM/SIGAPP Symposium on Applied Computing

March 2023

1932 pages

ISBN:9781450395175

DOI:10.1145/3555776

Conference Chairs:
Jiman Hong
Soongsil University, South Korea
,
Maart Lanperne
Tallinn University, Estonia
,
Program Chairs:
Juw Won Park
University of Louisville, USA
,
Tomas Cerny
Baylor University, USA
,
Publication Chair:
Hossain Shahriar
Kennesaw State University, USA

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGAPP: ACM Special Interest Group on Applied Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 June 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SAC '23

Sponsor:

SIGAPP

SAC '23: 38th ACM/SIGAPP Symposium on Applied Computing

March 27 - 31, 2023

Tallinn, Estonia

Acceptance Rates

Overall Acceptance Rate 1,650 of 6,669 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
51
Total Downloads

Downloads (Last 12 months)44
Downloads (Last 6 weeks)6

Reflects downloads up to 11 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Britikov KZlatkin IFedyukovich GAlt LSharygina N(2024)SolTG: A CHC-Based Solidity Test Case GeneratorComputer Aided Verification10.1007/978-3-031-65627-9_23(466-479)Online publication date: 26-Jul-2024
https://doi.org/10.1007/978-3-031-65627-9_23
Bombarda AGargantini A(2023)Incremental generation of combinatorial test suites starting from existing seed tests2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW58534.2023.00044(197-205)Online publication date: Apr-2023
https://doi.org/10.1109/ICSTW58534.2023.00044

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents