article

Testing across configurations: implications for combinatorial testing

Authors:

Gregg RothermelAuthors Info & Claims

ACM SIGSOFT Software Engineering Notes, Volume 31, Issue 6

Pages 1 - 9

https://doi.org/10.1145/1218776.1218785

Published: 01 November 2006 Publication History

Abstract

User configurable software systems allow users to customize functionality at run time. In essence, each such system consists of a family of potentially thousands or millions of program instantiations. Testing methods cannot test all of these configurations, therefore some sampling mechanism must be applied. A common approach to providing such a mechanism has been to use combinatorial interaction testing. To date, however, little work has been done to quantify the effects of different configurations on a test suites' operation and effectiveness. In this paper we present a case study that investigates the effects of changing configurations on two types of test suites. Our results show that test coverage and fault detection effectiveness do not vary much across configurations for entire test suites; however, for individual test cases and certain types of faults, configurations matter.

References

[1]

R. Brownlie, J. Prowse, and M. S. Phadke. Robust testing of AT&T PMX/StarMAIL using OATS. AT&T Technical Journal, 71(3):41--47, 1992.

[2]

K. Burr and W. Young. Combinatorial test techniques: Table-based automation, test generation and code coverage. In Proceedings of the International Conference on Software Testing Analysis & Review, 1998.

[3]

K. Burroughs, A. Jain, and R. L. Erickson. Improved quality of protocol testing through techniques of experimental design. In Supercomm/IC, IEEE International Conference on Communications, pages 745--752, 1994.

[4]

D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton. The AETG system: an approach to testing based on combinatorial design. IEEE Transactions on Software Engineering, 23(7):437--444, 1997.

Digital Library

[5]

D. M. Cohen, S. R. Dalal, J. Parelius, and G. C. Patton. The combinatorial design approach to automatic test generation. IEEE Software, 13(5):83--88, 1996.

Digital Library

[6]

S. R. Dalal, A. Jain, N. Karunanithi, J. M. Leaton, and C. M. Lott. Model-based testing of a highly programmable system. In Proceedings of the International Symposium on Software Reliability Engineering, pages 174--179, 1998.

Digital Library

[7]

S. R. Dalal, A. Jain, N. Karunanithi, J. M. Leaton, C. M. Lott, G. C. Patton, and B. M. Horowitz. Model-based testing in practice. In Proceedings of the International Conference on Software Engineering, pages 285--294, 1999.

Digital Library

[8]

S. R. Dalal, A. Jain, G. Patton, M. Rathi, and P. Seymour. AETGSM web: a Web based service for automatic efficient test generation from functional requirements. In Proceedings of the IEEE Workshop on Industrial Strength Formal Specification Techniques, pages 84--85, Oct 1998.

Digital Library

[9]

I. S. Dunietz, W. K. Ehrlich, B. D. Szablak, C. L. Mallows, and A. Iannino. Applying design of experiments to software testing. In Proceedings of the International Conference on Software Engineering, (ICSE), pages 205--215, 1997.

Digital Library

[10]

S. Elbaum, D. Gable, and G. Rothermel. The impact of software evolution on code coverage information. In International Conference on Software Maintenance, pages 169--179, 2001.

Digital Library

[11]

J. E. Forrester and B. P. Miller. An empirical study of the robustness of Windows NT applications using random testing. In USENIX Windows System Symposium, pages 1--10, July 2000.

Digital Library

[12]

M. Grindal, J. Offutt, and S. T. Andler. Combination testing strategies: A survey. Software Testing, Verification and Reliability, 15(3):167--199, 2005.

[13]

M. Hardojo, S. Elbaum, and Z. Wang. The effect of field data integrity on failure reproduction and fault isolation. In Workshop on Remote Analysis and Monitoring of Software Systems, pages 41--44, May 2004.

[14]

D. Kuhn and M. Reilly. An investigation of the applicability of design of experiments to software testing. In Proceedings of the NASA/IEEE Software Engineering Workshop, pages 91--95, 2002.

Digital Library

[15]

D. Kuhn, D. R. Wallace, and A. M. Gallo. Software fault interactions and implications for software testing. IEEE Transactions on Software Engineering, 30(6):418--421, 2004.

Digital Library

[16]

R. Mandl. Orthogonal Latin squares: An application of experiment design to compiler testing. Communications of the ACM, 28(10): 1054--1058, 1985.

Digital Library

[17]

A. Memon, A. Porter, C. Yilmaz, A. Nagarajan, D. Schmidt, and B. Natarajan. Skoll: Distributed continuous quality assurance. In Proceedings of the International Conference on Software Engineering, pages 459--468, May 2004.

Digital Library

[18]

B. Miller, G. Cooksey, and F. Moore. An empirical study of the robustness of MacOS applications using random testing. In First International Workshop on Random Testing, pages 46--54, July 2006.

Digital Library

[19]

T. J. Ostrand and M. J. Balcer. The category-partition method for specifying and generating functional tests. Communications of the ACM, 31:678--686, 1988.

Digital Library

[20]

G. Rothermel and M. J. Harrold. Analyzing regression test selection techniques. IEEE Transactions on Software Engineering, 22(8):529--551, Aug. 1996.

Digital Library

[21]

E. J. Weyuker. Testing component-based software: A cautionary tale. IEEE Software, 15(5):54--59, 1998.

Digital Library

[22]

C. Yilmaz, M. B. Cohen, and A. Porter. Covering arrays for efficient fault characterization in complex configuration spaces. In International Symposium on Software Testing and Analysis, pages 45--54, July 2004.

Digital Library

[23]

C. Yilmaz, M. B. Cohen, and A. Porter. Covering arrays for efficient fault characterization in complex configuration spaces. IEEE Transactions on Software Engineering, 31(1):20--34, Jan 2006.

Digital Library

[24]

C. You. My IE3.1. http://www.myie2.com/html_en/home.htm.

Cited By

Andrew CAnn Furman DRawlins RGildein MFarchi E(2024)A Novel Double T-wise Approach to Root Cause Analysis via Combinatorial Test Set Completion2024 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW60967.2024.00045(193-196)Online publication date: 27-May-2024
https://doi.org/10.1109/ICSTW60967.2024.00045
Nie PBanerjee RLi JMooney RGligoric MGrundy JPollock LPenta M(2023)Learning Deep Semantics for Test CompletionProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00178(2111-2123)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE48619.2023.00178
Wang SLian XMarinov DXu TGrundy JPollock LPenta M(2023)Test Selection for Unified Regression TestingProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00145(1687-1699)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE48619.2023.00145
Show More Cited By

Index Terms

Testing across configurations: implications for combinatorial testing
1. Software and its engineering

Recommendations

Configuration-aware regression testing: an empirical study of sampling and prioritization
ISSTA '08: Proceedings of the 2008 international symposium on Software testing and analysis

Configurable software lets users customize applications in many ways, and is becoming increasingly prevalent. Researchers have created techniques for testing configurable software, but to date, only a little research has addressed the problems of ...
Prioritizing Variable-Strength Covering Array
COMPSAC '13: Proceedings of the 2013 IEEE 37th Annual Computer Software and Applications Conference

Combinatorial interaction testing is a well-studied testing strategy, and has been widely applied in practice. Combinatorial interaction test suite, such as fixed-strength and variable-strength interaction test suite, is widely used for combinatorial ...
Measuring and Mitigating Gaps in Structural Testing
ICSE '23: Proceedings of the 45th International Conference on Software Engineering

Structural code coverage is a popular test adequacy metric that measures the percentage of program structure (e.g., statement, branch, decision) executed by a test suite. While structural coverage has several benefits, previous studies suggested that ...

Comments

Information & Contributors

Information

Published In

cover image ACM SIGSOFT Software Engineering Notes

ACM SIGSOFT Software Engineering Notes Volume 31, Issue 6

November 2006

182 pages

ISSN:0163-5948

DOI:10.1145/1218776

Issue’s Table of Contents

Copyright © 2006 Authors.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 November 2006

Published in SIGSOFT Volume 31, Issue 6

Check for updates

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

59
Total Citations
View Citations
579
Total Downloads

Downloads (Last 12 months)12
Downloads (Last 6 weeks)3

Reflects downloads up to 15 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Andrew CAnn Furman DRawlins RGildein MFarchi E(2024)A Novel Double T-wise Approach to Root Cause Analysis via Combinatorial Test Set Completion2024 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW60967.2024.00045(193-196)Online publication date: 27-May-2024
https://doi.org/10.1109/ICSTW60967.2024.00045
Nie PBanerjee RLi JMooney RGligoric MGrundy JPollock LPenta M(2023)Learning Deep Semantics for Test CompletionProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00178(2111-2123)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE48619.2023.00178
Wang SLian XMarinov DXu TGrundy JPollock LPenta M(2023)Test Selection for Unified Regression TestingProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00145(1687-1699)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE48619.2023.00145
Mordahl AZhang ZSoles DWei SGrundy JPollock LPenta M(2023)ECSTATIC: An Extensible Framework for Testing and Debugging Configurable Static AnalysisProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00056(550-562)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE48619.2023.00056
Ma'ayan DMaoz SRozi RSyriani ESahraoui H(2022)Validating the correctness of reactive systems specifications through systematic explorationProceedings of the 25th International Conference on Model Driven Engineering Languages and Systems10.1145/3550355.3552425(132-142)Online publication date: 23-Oct-2022
https://dl.acm.org/doi/10.1145/3550355.3552425
Tzoref-Brill RSinha SNassar AGoldin VKermany H(2022)TackleTest: A Tool for Amplifying Test Generation via Type-Based Combinatorial Coverage2022 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST53961.2022.00050(444-455)Online publication date: Apr-2022
https://doi.org/10.1109/ICST53961.2022.00050
Mahdavi-Hezaveh RAjmeri NWilliams L(2022)Feature toggles as codeInformation and Software Technology10.1016/j.infsof.2021.106813145:COnline publication date: 1-May-2022
https://dl.acm.org/doi/10.1016/j.infsof.2021.106813
Shang WZhang GWang TZhang R(2021)A Test Cases Generation Method for Industrial Control Protocol TestScientific Programming10.1155/2021/66117322021Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.1155/2021/6611732
Fischer S(2021)A Case Study on the Evolution of Configuration Options of a Highly-Configurable Software System2021 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER50967.2021.00079(630-635)Online publication date: Mar-2021
https://doi.org/10.1109/SANER50967.2021.00079
Niu XNie CLeung HLei YWang XXu JWang Y(2020)An Interleaving Approach to Combinatorial Testing and Failure-Inducing Interaction IdentificationIEEE Transactions on Software Engineering10.1109/TSE.2018.286577246:6(584-615)Online publication date: 1-Jun-2020
https://dl.acm.org/doi/10.1109/TSE.2018.2865772
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents