article

IncLing: efficient product-line testing using incremental pairwise sampling

Authors:

Mustafa Al-Hajjaji,

Sebastian Krieter,

Gunter SaakeAuthors Info & Claims

ACM SIGPLAN Notices, Volume 52, Issue 3

Pages 144 - 155

https://doi.org/10.1145/3093335.2993253

Published: 20 October 2016 Publication History

Abstract

A software product line comprises a family of software products that share a common set of features. It enables customers to compose software systems from a managed set of features. Testing every product of a product line individually is often infeasible due to the exponential number of possible products in the number of features. Several approaches have been proposed to restrict the number of products to be tested by sampling a subset of products achieving sufficient combinatorial interaction coverage. However, existing sampling algorithms do not scale well to large product lines, as they require a considerable amount of time to generate the samples. Moreover, samples are not available until a sampling algorithm completely terminates. As testing time is usually limited, we propose an incremental approach of product sampling for pairwise interaction testing (called IncLing), which enables developers to generate samples on demand in a step-wise manner. Furthermore, IncLing uses heuristics to efficiently achieve pairwise interaction coverage with a reasonable number of products. We evaluated IncLing by comparing it against existing sampling algorithms using feature models of different sizes. The results of our approach indicate efficiency improvements for product-line testing.

References

[1]

M. Al-Hajjaji. Scalable Sampling and Prioritization for Product-Line Testing. In Proc. Software Engineering (SE), pages 295–298. Gesellschaft für Informatik (GI), 2015.

[2]

M. Al-Hajjaji, J. Meinicke, S. Krieter, R. Schröter, T. Thüm, T. Leich, and G. Saake. Tool Demo: Testing Configurable Systems with FeatureIDE. In Proc. Int’l Conf. Generative Programming and Component Engineering (GPCE), 2016. To appear.

Digital Library

[3]

M. Al-Hajjaji, T. Thüm, J. Meinicke, M. Lochau, and G. Saake. Similarity-Based Prioritization in Software Product-Line Testing. In Proc. Int’l Software Product Line Conf. (SPLC), pages 197–206. ACM, 2014.

Digital Library

[4]

S. Apel, D. Batory, C. Kästner, and G. Saake. Feature-Oriented Software Product Lines: Concepts and Implementation. Springer, 2013.

Digital Library

[5]

D. Batory, J. N. Sarvela, and A. Rauschmayer. Scaling Step-Wise Refinement. IEEE Trans. Software Engineering (TSE), 30(6):355–371, 2004.

Digital Library

[6]

D. Benavides, S. Segura, and A. Ruiz-Cortés. Automated Analysis of Feature Models 20 Years Later: A Literature Review. Information Systems, 35(6):615–708, 2010.

Digital Library

[7]

I. D. Carmo Machado, J. D. McGregor, Y. a. C. Cavalcanti, and E. S. De Almeida. On Strategies for Testing Software Product Lines: A Systematic Literature Review. J. Information and Software Technology (IST), 56(10):1183–1199, 2014.

Digital Library

[8]

V. Chvatal. A Greedy Heuristic for the Set-Covering Problem. Mathematics of operations research, 4(3):233– 235, 1979.

Digital Library

[9]

P. Clements and L. Northrop. Software Product Lines: Practices and Patterns. Addison-Wesley, 2001.

Digital Library

[10]

M. B. Cohen, M. B. Dwyer, and J. Shi. Coverage and Adequacy in Software Product Line Testing. In Proc. Int’l Symposium in Software Testing and Analysis (ISSTA), pages 53–63. ACM, 2006.

Digital Library

[11]

M. B. Cohen, M. B. Dwyer, and J. Shi. Interaction Testing of Highly-Configurable Systems in the Presence of Constraints. In Proc. Int’l Symposium in Software Testing and Analysis (ISSTA), pages 129–139. ACM, 2007.

Digital Library

[12]

S. A. Cook. The Complexity of Theorem-proving Procedures. In Proc. of ACM Symposium on Theory of Computing (STOC), pages 151–158. ACM, 1971.

Digital Library

[13]

K. Czarnecki and U. Eisenecker. Generative Programming: Methods, Tools, and Applications. ACM/Addison-Wesley, 2000.

Digital Library

[14]

L. Engebretsen. The Nonapproximability of Non-Boolean Predicates. SIAM Journal on Discrete Mathematics (JDM), 18(1):114–129, 2005.

Digital Library

[15]

B. J. Garvin, M. B. Cohen, and M. B. Dwyer. Evaluating Improvements to a Meta-Heuristic Search for Constrained Interaction Testing. Empirical Software Engineering (EMSE), 16(1):61–102, 2011.

Digital Library

[16]

C. Henard, M. Papadakis, G. Perrouin, J. Klein, P. Heymans, and Y. L. Le Traon. Bypassing the Combinatorial Explosion: Using Similarity to Generate and Prioritize T-Wise Test Configurations for Software Product Lines. IEEE Trans. Software Engineering (TSE), 40(7):650– 670, 2014.

Digital Library

[17]

M. Jackson and P. Zave. Distributed Feature Composition: A Virtual Architecture for Telecommunications Services. IEEE Trans. Software Engineering (TSE), 24(10):831–847, 1998.

Digital Library

[18]

M. F. Johansen, Ø. Haugen, and F. Fleurey. Properties of Realistic Feature Models Make Combinatorial Testing of Product Lines Feasible. In Proc. Int’l Conf. Model Driven Engineering Languages and Systems (MODELS), pages 638–652. Springer, 2011.

Digital Library

[19]

M. F. Johansen, Ø. Haugen, and F. Fleurey. An Algorithm for Generating T-Wise Covering Arrays from Large Feature Models. In Proc. Int’l Software Product Line Conf. (SPLC), pages 46–55. ACM, 2012.

Digital Library

[20]

K. C. Kang, S. G. Cohen, J. A. Hess, W. E. Novak, and A. S. Peterson. Feature-Oriented Domain Analysis (FODA) Feasibility Study. Technical Report CMU/SEI- 90-TR-21, Software Engineering Institute, 1990.

[21]

C. H. P. Kim, D. Batory, and S. Khurshid. Reducing Combinatorics in Testing Product Lines. In Proc. Int’l Conf. Aspect-Oriented Software Development (AOSD), pages 57—68. ACM, 2011.

Digital Library

[22]

M. Kowal, S. Schulze, and I. Schaefer. Towards Efficient SPL Testing by Variant Reduction. In Proce. Int’l workshop on Variability & composition (VariComp), pages 1–6. ACM, 2013.

Digital Library

[23]

D. R. Kuhn, D. R. Wallace, and A. M. Gallo Jr. Software Fault Interactions and Implications for Software Testing. IEEE Trans. Software Engineering (TSE), 30(6):418– 421, 2004.

Digital Library

[24]

Y. Lei, R. N. Kacker, D. R. Kuhn, V. Okun, and J. Lawrence. IPOG: A General Strategy for T-Way Software Testing. In Proc. Int’l Conf. Engineering of Computer-Based Systems (ECBS), pages 549–556. IEEE, 2007.

Digital Library

[25]

J. Liebig, A. von Rhein, C. Kästner, S. Apel, J. Dörre, and C. Lengauer. Scalable Analysis of Variable Software. In Proc. Europ. Software Engineering Conf./Foundations of Software Engineering (ESEC/FSE), pages 81–91. ACM, 2013.

Digital Library

[26]

M. Lochau, I. Schaefer, J. Kamischke, and S. Lity. Incremental Model-Based Testing of Delta-oriented Software Product Lines. In Proc. of Int’l Conf. on Tests and Proofs (TAP), pages 67–82. Springer, 2012.

Digital Library

[27]

F. Medeiros, C. Kästner, M. Ribeiro, R. Gheyi, and S. Apel. A Comparison of 10 Sampling Algorithms for Configurable Systems. In Proc. Int’l Conf. Software Engineering (ICSE), ICSE ’16, pages 643–654. ACM, 2016.

Digital Library

[28]

J. Melo, E. Flesborg, C. Brabrand, and A. Wasowski. A Quantitative Analysis of Variability Warnings in Linux. In Proc. Int’l Workshop Variability Modelling of Software-Intensive Systems (VaMoS), VaMoS ’16, pages 3–8. ACM, 2016.

Digital Library

[29]

S. Oster, F. Markert, and P. Ritter. Automated Incremental Pairwise Testing of Software Product Lines. In Proc. Int’l Software Product Line Conf. (SPLC), pages 196–210. Springer, 2010.

Digital Library

[30]

S. Oster, I. Zorcic, F. Markert, and M. Lochau. MoSo-PoLiTe - Tool Support for Pairwise and Model-Based Software Product Line Testing. In Proc. Int’l Workshop Variability Modelling of Software-Intensive Systems (VaMoS), pages 79–82. ACM, 2011.

Digital Library

[31]

G. Perrouin, S. Oster, S. Sen, J. Klein, B. Baudry, and Y. Le Traon. Pairwise Testing for Software Product Lines: Comparison of Two Approaches. Software Quality Journal (SQJ), 20(3-4):605–643, 2012.

Digital Library

[32]

G. Perrouin, S. Sen, J. Klein, B. Baudry, and Y. Le Traon. Automated and Scalable T-Wise Test Case Generation Strategies for Software Product Lines. In Proc. Int’l Conf. Software Testing, Verification and Validation (ICST), pages 459–468. IEEE, 2010.

Digital Library

[33]

K. Pohl, G. Böckle, and F. J. van der Linden. Software Product Line Engineering: Foundations, Principles and Techniques. Springer, 2005.

Digital Library

[34]

J. Shi, M. B. Cohen, and M. B. Dwyer. Integration Testing of Software Product Lines Using Compositional Symbolic Execution. In Proc. Int’l Conf. Fundamental Approaches to Software Engineering (FASE), pages 270– 284. Springer, 2012.

Digital Library

[35]

T. Thüm, C. Kästner, F. Benduhn, J. Meinicke, G. Saake, and T. Leich. FeatureIDE: An Extensible Framework for Feature-Oriented Software Development. Science of Computer Programming (SCP), 79(0):70–85, 2014.

Digital Library

[36]

D. M. Weiss. The Product Line Hall of Fame. In Proc. Int’l Software Product Line Conf. (SPLC), page 395. IEEE, 2008.

Digital Library

Cited By

Marinho EPessoa Neto IFerreira FDiniz JFigueiredo E(2024)RIFDiscoverer: A Tool for Finding Resource Interaction FailuresProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3676603(30-33)Online publication date: 2-Sep-2024
https://dl.acm.org/doi/10.1145/3646548.3676603
Baranov EChakraborty SLegay AMeel KVinodchandran N(2024)A Scalable t-Wise Coverage Estimator: Algorithms and ApplicationsIEEE Transactions on Software Engineering10.1109/TSE.2024.341991950:8(2021-2039)Online publication date: Aug-2024
https://doi.org/10.1109/TSE.2024.3419919
Agh HAzamnouri AWagner S(2024)Software product line testing: a systematic literature reviewEmpirical Software Engineering10.1007/s10664-024-10516-x29:6Online publication date: 2-Sep-2024
https://doi.org/10.1007/s10664-024-10516-x
Show More Cited By

Index Terms

IncLing: efficient product-line testing using incremental pairwise sampling
1. General and reference
  1. Cross-computing tools and techniques
    1. Metrics
2. Software and its engineering
  1. Software creation and management
    1. Software development techniques
      1. Reusability
  2. Software notations and tools
    1. Software libraries and repositories

Recommendations

IncLing: efficient product-line testing using incremental pairwise sampling
GPCE 2016: Proceedings of the 2016 ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences

A software product line comprises a family of software products that share a common set of features. It enables customers to compose software systems from a managed set of features. Testing every product of a product line individually is often ...
Effective product-line testing using similarity-based product prioritization

A software product line comprises a family of software products that share a common set of features. Testing an entire product-line product-by-product is infeasible due to the potentially exponential number of products in the number of features. ...
Similarity-based prioritization in software product-line testing
SPLC '14: Proceedings of the 18th International Software Product Line Conference - Volume 1

Exhaustively testing every product of a software product line (SPL) is a difficult task due to the combinatorial explosion of the number of products. Combinatorial interaction testing is a technique to reduce the number of products under test. However, ...

Comments

Information & Contributors

Information

Published In

cover image ACM SIGPLAN Notices

ACM SIGPLAN Notices Volume 52, Issue 3

GPCE '16

March 2017

212 pages

ISSN:0362-1340

EISSN:1558-1160

DOI:10.1145/3093335

Editor:
Matthew Fluet

Issue’s Table of Contents

GPCE 2016: Proceedings of the 2016 ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences
October 2016
212 pages
ISBN:9781450344463
DOI:10.1145/2993236
General Chair:
Bernd Fischer
Stellenbosch University, South Africa
,
Program Chair:
Ina Schaefer
TU Braunschweig, Germany

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 October 2016

Published in SIGPLAN Volume 52, Issue 3

Check for updates

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

65
Total Citations
View Citations
370
Total Downloads

Downloads (Last 12 months)30
Downloads (Last 6 weeks)4

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Marinho EPessoa Neto IFerreira FDiniz JFigueiredo E(2024)RIFDiscoverer: A Tool for Finding Resource Interaction FailuresProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3676603(30-33)Online publication date: 2-Sep-2024
https://dl.acm.org/doi/10.1145/3646548.3676603
Baranov EChakraborty SLegay AMeel KVinodchandran N(2024)A Scalable t-Wise Coverage Estimator: Algorithms and ApplicationsIEEE Transactions on Software Engineering10.1109/TSE.2024.341991950:8(2021-2039)Online publication date: Aug-2024
https://doi.org/10.1109/TSE.2024.3419919
Agh HAzamnouri AWagner S(2024)Software product line testing: a systematic literature reviewEmpirical Software Engineering10.1007/s10664-024-10516-x29:6Online publication date: 2-Sep-2024
https://doi.org/10.1007/s10664-024-10516-x
Bombarda ABonfanti SGargantini A(2023)On the Reuse of Existing Configurations for Testing Evolving Feature ModelsProceedings of the 27th ACM International Systems and Software Product Line Conference - Volume B10.1145/3579028.3609017(67-76)Online publication date: 28-Aug-2023
https://dl.acm.org/doi/10.1145/3579028.3609017
Eichhorn DPett TPrzigoda NKindsvater JSeidl CSchaefer I(2023)Coverage-Driven Test Automation for Highly-Configurable Railway SystemsProceedings of the 17th International Working Conference on Variability Modelling of Software-Intensive Systems10.1145/3571788.3571794(23-30)Online publication date: 25-Jan-2023
https://dl.acm.org/doi/10.1145/3571788.3571794
Gwasem IDu WMcAllister A(2023)Goal-Oriented Prioritized Non-Functional Testing with Stakeholders' Priorities2023 IEEE 8th International Conference on Engineering Technologies and Applied Sciences (ICETAS)10.1109/ICETAS59148.2023.10346293(1-8)Online publication date: 25-Oct-2023
https://doi.org/10.1109/ICETAS59148.2023.10346293
Abolfazli ASpiegelberg JPalmer GAnand A(2023)A Deep Reinforcement Learning Approach to Configuration Sampling Problem2023 IEEE International Conference on Data Mining (ICDM)10.1109/ICDM58522.2023.00009(1-10)Online publication date: 1-Dec-2023
https://doi.org/10.1109/ICDM58522.2023.00009
Marinho EFerreira FDiniz JFigueiredo E(2023)Evaluating testing strategies for resource related failures in mobile applicationsSoftware Quality Journal10.1007/s11219-023-09627-331:4(1121-1147)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1007/s11219-023-09627-3
Jung PKang SLee J(2022)Reducing Redundant Test Executions in Software Product Line Testing—A Case StudyElectronics10.3390/electronics1107116511:7(1165)Online publication date: 6-Apr-2022
https://doi.org/10.3390/electronics11071165
Birkemeyer LPett TVogelsang ASeidl CSchaefer I(2022)Feature-Interaction Sampling for Scenario-based Testing of Advanced Driver Assistance Systems✱Proceedings of the 16th International Working Conference on Variability Modelling of Software-Intensive Systems10.1145/3510466.3510474(1-10)Online publication date: 23-Feb-2022
https://dl.acm.org/doi/10.1145/3510466.3510474
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents