research-article

Open access

Constrained detecting arrays for fault localization in combinatorial testing

Authors:

Tatsuhiro TsuchiyaAuthors Info & Claims

SAC '20: Proceedings of the 35th Annual ACM Symposium on Applied Computing

Pages 1971 - 1978

https://doi.org/10.1145/3341105.3373952

Published: 30 March 2020 Publication History

Abstract

Detecting Arrays (DAs) are mathematical objects that enable fault localization in combinatorial interaction testing. Each row of a DA serves as a test case, whereas a whole DA is treated as a test suite. In real-world testing problems, it is often the case that some constraints exist among test parameters. In this paper, we show that it may be impossible to construct a DA using only constraint-satisfying test cases. The reason for this is that a set of some faulty interactions may always mask the effect of other faulty interactions in the presence of constraints. Based on this observation, we propose the notion of Constrained Detecting Arrays (CDAs) to adapt DAs to practical situations. The definition of CDAs requires that all rows of a CDA must satisfy the constraints and the same fault localization capability as the DA must hold except for such inherently undetectable faults. We then propose a computational method for constructing CDAs. Experimental results obtained by using a program that implements the method show that the method was able to produce CDAs within a reasonable time for practical problem instances.

References

[1]

David M. Cohen, Siddhartha R. Dalal, Michael L. Fredman, and Gardner C. Patton. 1997. The AETG System: An Approach to Testing Based on Combinatiorial Design. IEEE Trans. Software Eng. 23, 7 (1997), 437--444.

Digital Library

[2]

M. B. Cohen, M. B. Dwyer, and J. Shi. 2008. Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach. IEEE Trans. Software Eng. 34, 5 (Sept 2008), 633--650.

Digital Library

[3]

Charles J. Colbourn and Daniel W. McClary. 2008. Locating and detecting arrays for interaction faults. Journal of combinatorial optimization 15, 1 (2008), 17--48.

[4]

Charles J. Colbourn and Violet R. Syrotiuk. 2019. Detecting Arrays for Main Effects. In Algebraic Informatics - 8th International Conference, CAI 2019, Niš, Serbia, June 30 - July 4, 2019, Proceedings. 112--123.

[5]

Leonardo De Moura and Nikolaj Bjørner. 2008. Z3: An Efficient SMT Solver. In Proceedings of the Theory and Practice of Software, 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS'08/ETAPS'08). Springer-Verlag, Berlin, Heidelberg, 337--340.

Digital Library

[6]

Philippe Galinier, Segla Kpodjedo, and Giulio Antoniol. 2017. A Penalty-based Tabu Search for Constrained Covering Arrays. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO '17). ACM, New York, NY, USA, 1288--1294.

Digital Library

[7]

A. Gargantini and M. Radavelli. 2018. Migrating Combinatorial Interaction Test Modeling and Generation to the Web. In 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). 308--317.

[8]

B. J. Garvin, M. B. Cohen, and M. B. Dwyer. 2009. An Improved Meta-heuristic Search for Constrained Interaction Testing. In 2009 1st International Symposium on Search Based Software Engineering. 13--22.

Digital Library

[9]

Brady J. Garvin, Myra B. Cohen, and Matthew B. Dwyer. 2011. Evaluating improvements to a meta-heuristic search for constrained interaction testing. Empirical Software Engineering 16, 1 (2011), 61--102.

Digital Library

[10]

Brahim Hnich, Steven Prestwich, and Evgeny Selensky. 2005. Constraint-Based Approaches to the Covering Test Problem. In Recent Advances in Constraints, Boi V. Faltings, Adrian Petcu, François Fages, and Francesca Rossi (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 172--186.

[11]

H. Jin, T. Kitamura, E. Choi, and T. Tsuchiya. 2018. A Satisfiability-Based Approach to Generation of Constrained Locating Arrays. In 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). 285--294.

[12]

Hao Jin and Tatsuhiro Tsuchiya. 2018. Constrained locating arrays for combinatorial interaction testing. CoRR abs/1801.06041 (2018), 1--12. arXiv:1801.06041 http://arxiv.org/abs/1801.06041

[13]

Martin Fagereng Johansen, Øystein Haugen, Franck Fleurey, Anne Grete Eldegard, and Torbjørn Syversen. 2012. Generating Better Partial Covering Arrays by Modeling Weights on Sub-product Lines. In Model Driven Engineering Languages and Systems, Robert B. France, Jürgen Kazmeier, Ruth Breu, and Colin Atkinson (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 269--284.

[14]

T. Konishi, H. Kojima, H. Nakagawa, and T. Tsuchiya. 2017. Finding Minimum Locating Arrays Using a SAT Solver. In 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops. 276--277.

[15]

D. Richard Kuhn, Raghu N. Kacker, and Yu Lei. 2013. Introduction to combinatorial testing. CRC Press.

[16]

D. Richard Kuhn and Daniel R. Wallace. 2004. Software fault interactions and implications for software testing. IEEE Transactions on Software Engineering 30, 6 (June 2004), 418--421.

[17]

Yu Lei, Raghu Kacker, D. Richard Kuhn, Vadim Okun, and James Lawrence. 2008. IPOG/IPOG-D: efficient test generation for multi-way combinatorial testing. Software Testing, Verification and Reliability 18, 3 (2008), 125--148. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/stvr.381

Digital Library

[18]

Toru Nanba, Tatsuhiro Tsuchiya, and Tohru Kikuno. 2012. Using Satisfiability Solving for Pairwise Testing in the Presence of Constraints. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences E95.A, 9 (2012), 1501--1505.

[19]

Changhai Nie and Hareton Leung. 2011. The Minimal Failure-Causing Schema of Combinatorial Testing. ACM Trans. Softw. Eng. Methodol. 20, 4, Article 15 (Sept. 2011), 38 pages.

Digital Library

[20]

Changhai Nie and Hareton Leung. 2011. A Survey of Combinatorial Testing. ACM Comput. Surv. 43, 2, Article 11 (Feb. 2011), 29 pages.

Digital Library

[21]

Itai Segall, Rachel Tzoref-Brill, and Eitan Farchi. 2011. Using binary decision diagrams for combinatorial test design. In Proc. of the 2011 International Symposium on Software Testing and Analysis (ISSTA). ACM, 254--264.

Digital Library

[22]

Stephen A. Seidel, Kaushik Sarkar, Charles J. Colbourn, and Violet R. Syrotiuk. 2018. Separating Interaction Effects Using Locating and Detecting Arrays. In Combinatorial Algorithms - 29th International Workshop, IWOCA 2018, Singapore, July 16--19, 2018, Proceedings. 349--360.

[23]

Ce Shi, Yu Tang, and Jianxing Yin. 2014. Optimum mixed level detecting arrays. The Annals of Statistics 42 (08 2014).

[24]

Ce Shi and Cheng Min Wang. 2016. Optimum detecting arrays for independent interaction faults. Acta Mathematica Sinica, English Series 32, 2 (01 Feb 2016), 199--212.

[25]

Yu Tang and Jian Xing Yin. 2011. Detecting arrays and their optimality. Acta Mathematica Sinica, English Series 27, 12 (01 Dec 2011), 2309--2318.

[26]

Tatsuhiro Tsuchiya. 2019. Using binary decision diagrams for constraint handling in combinatorial interaction testing. CoRR abs/1907.01779 (2019). arXiv:1907.01779 http://arxiv.org/abs/1907.01779

[27]

Huayao Wu, Nie Changhai, Justyna Petke, Yue Jia, and Mark Harman. 2019. Comparative Analysis of Constraint Handling Techniques for Constrained Combinatorial Testing. IEEE Transactions on Software Engineering PP (11 2019), 1--1.

[28]

Huayao Wu, Changhai Nie, Justyna Petke, Yue Jia, and Mark Harman. 2019. A Survey of Constrained Combinatorial Testing. CoRR abs/1908.02480 (2019). arXiv:1908.02480 http://arxiv.org/abs/1908.02480

Cited By

Kampel LWagner MSimos DNica MDodig DKaufmann DWotawa F(2023)Applying CT-FLA for AEB Function Testing: A Virtual Driving Case Study2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW58534.2023.00049(237-245)Online publication date: Apr-2023
https://doi.org/10.1109/ICSTW58534.2023.00049
Jin HShi CTsuchiya T(2023)Constrained detecting arraysInformation and Software Technology10.1016/j.infsof.2022.107045153:COnline publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1016/j.infsof.2022.107045
Jin HTsuchiya T(2020)A Two-Step Heuristic Algorithm for Generating Constrained Detecting Arrays for Combinatorial Interaction Testing2020 IEEE 29th International Conference on Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE)10.1109/WETICE49692.2020.00050(219-224)Online publication date: Sep-2020
https://doi.org/10.1109/WETICE49692.2020.00050

Index Terms

Constrained detecting arrays for fault localization in combinatorial testing
1. Mathematics of computing
  1. Discrete mathematics
    1. Combinatorics
      1. Combinatorial optimization
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Constrained detecting arrays: Mathematical structures for fault identification in combinatorial interaction testing
Abstract Context:
Detecting arrays are mathematical structures aimed at fault identification in combinatorial interaction testing. However, they cannot be directly applied to systems that have constraints on the test parameters. ...
Highlights
- Constrained detecting arrays (CDAs) are proposed for identification of faulty interactions in combinatorial testing.
Tester Feedback Driven Fault Localization
ICST '12: Proceedings of the 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation

Coincidentally correct test cases are those that execute faulty statements but do not cause failures. Such test cases reduce the effectiveness of spectrum-based fault localization techniques, such as Ochiai, because the correlation of failure with the ...
Test Case-Aware Combinatorial Interaction Testing

The configuration spaces of modern software systems are too large to test exhaustively. Combinatorial interaction testing (CIT) approaches, such as covering arrays, systematically sample the configuration space and test only the selected configurations ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SAC '20: Proceedings of the 35th Annual ACM Symposium on Applied Computing

March 2020

2348 pages

ISBN:9781450368667

DOI:10.1145/3341105

Conference Chairs:
Chih-Cheng Hung
Kennesaw State University
,
Tomas Cerny
Baylor University
,
Program Chairs:
Dongwan Shin
New Mexico Tech
,
Alessio Bechini
University of Pisa, Italy

Copyright © 2020 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGAPP: ACM Special Interest Group on Applied Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 March 2020

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SAC '20

Sponsor:

SIGAPP

SAC '20: The 35th ACM/SIGAPP Symposium on Applied Computing

March 30 - April 3, 2020

Brno, Czech Republic

Acceptance Rates

Overall Acceptance Rate 1,650 of 6,669 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
390
Total Downloads

Downloads (Last 12 months)52
Downloads (Last 6 weeks)10

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kampel LWagner MSimos DNica MDodig DKaufmann DWotawa F(2023)Applying CT-FLA for AEB Function Testing: A Virtual Driving Case Study2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW58534.2023.00049(237-245)Online publication date: Apr-2023
https://doi.org/10.1109/ICSTW58534.2023.00049
Jin HShi CTsuchiya T(2023)Constrained detecting arraysInformation and Software Technology10.1016/j.infsof.2022.107045153:COnline publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1016/j.infsof.2022.107045
Jin HTsuchiya T(2020)A Two-Step Heuristic Algorithm for Generating Constrained Detecting Arrays for Combinatorial Interaction Testing2020 IEEE 29th International Conference on Enabling Technologies: Infrastructure for Collaborative Enterprises (WETICE)10.1109/WETICE49692.2020.00050(219-224)Online publication date: Sep-2020
https://doi.org/10.1109/WETICE49692.2020.00050

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents