short-paper

Public Access

t-wise Coverage by Uniform Sampling

Authors:

Don BatoryAuthors Info & Claims

SPLC '19: Proceedings of the 23rd International Systems and Software Product Line Conference - Volume A

Pages 84 - 87

https://doi.org/10.1145/3336294.3342359

Published: 09 September 2019 Publication History

Abstract

Efficiently testing large configuration spaces of Software Product Lines (SPLs) needs a sampling algorithm that is both scalable and provides good t-wise coverage. The 2019 SPLC Sampling Challenge provides large real-world feature models and asks for a t-wise sampling algorithm that can work for those models.

We evaluated t-wise coverage by uniform sampling (US) the configurations of one of the provided feature models. US means that every (legal) configuration is equally likely to be selected. US yields statistically representative samples of a configuration space and can be used as a baseline to compare other sampling algorithms.

We used existing algorithm called Smarch to uniformly sample SPL configurations. While uniform sampling alone was not enough to produce 100% 1-wise and 2-wise coverage, we used standard probabilistic analysis to explain our experimental results and to conjecture how uniform sampling may enhance the scalability of existing t-wise sampling algorithms.

References

[1]

Mustafa Al-Hajjaji, Sebastian Krieter, Thomas Thüm, Malte Lochau, and Gunter Saake. 2016. IncLing: efficient product-line testing using incremental pairwise sampling. In ACM SIGPLAN Notices. ACM, ACM, New York, NY, USA, 144--155.

Digital Library

[2]

Mustafa Al-Hajjaji, Thomas Thüm, Malte Lochau, Jens Meinicke, and Gunter Saake. 2019. Effective product-line testing using similarity-based product prioritization. Software & Systems Modeling 18, 1 (2019), 499--521.

Digital Library

[3]

Sven Apel, Don Batory, Christian Kaestner, and Gunter Saake. 2013. Feature-Oriented Software Product Lines. Springer, Berlin, Heidelberg.

Digital Library

[4]

Don Batory. 2005. Feature models, grammars, and propositional formulas. In International Conference on Software Product Lines. Springer, Springer, NY, USA, 7--20.

Digital Library

[5]

Armin Biere, Marijn Heule, and Hans van Maaren. 2009. Handbook of satisfiability. Vol. 185. IOS press, IEEE.

[6]

Brady J Garvin, Myra B Cohen, and Matthew B Dwyer. 2011. Evaluating improvements to a meta-heuristic search for constrained interaction testing. Empirical Software Engineering 16, 1 (2011), 61--102.

Digital Library

[7]

C.M. Grinstead and J.L. Snell. 2019. Intro to Probability - Dartmouth College. https://www.dartmouth.edu/~chance/teaching_aids/books_articles/probability_book/amsbook.mac.pdf.

[8]

Jeho Oh, Paul Gazzillo, Don Batory, Marijn Heule, and Maggie Myers. 2019. Uniform Sampling from Kconfig Feature Models. Technical Report TR-19-02. University of Texas at Austin, Department of Computer Science.

[9]

Martin Fagereng Johansen, Øystein Haugen, and Franck Fleurey. 2011. Properties of realistic feature models make combinatorial testing of product lines feasible. In International Conference on Model Driven Engineering Languages and Systems. Springer, Springer, Berlin, Heidelberg, 638--652.

Digital Library

[10]

Martin Fagereng Johansen, Øystein Haugen, Franck Fleurey, Anne Grete Eldegard, and Torbjørn Syversen. 2012. Generating better partial covering arrays by modeling weights on sub-product lines. In International Conference on Model Driven Engineering Languages and Systems. Springer, Springer, Berlin, Heidelberg, 269--284.

Digital Library

[11]

Christian Kaltenecker, Alexander Grebhahn, Norbert Siegmund, Jianmei Guo, and Sven Apel. 2019. Distance-Based Sampling of Software Configuration Spaces. In Proceedings of the 2019 International Conference on Software Engineering. ICSE, IEEE/ACM, Piscataway, NJ, USA, 0.

Digital Library

[12]

Jeho Oh, Don Batory, Margaret Myers, and Norbert Siegmund. 2017. Finding near-optimal configurations in product lines by random sampling. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. ACM, IEEE/ACM, Piscataway, NJ, USA, 61--71.

Digital Library

[13]

Quentin Plazar, Mathieu Acher, Gilles Perrouin, Xavier Devroey, and Maxime Cordy. 2019. Uniform Sampling of SAT Solutions for Configurable Systems: Are We There Yet?. In 12th International Conference on Software Testing, Verification, and Validation ICST 2019. IEEE, Piscataway, NJ, USA.

[14]

Thomas Thüm, Christian Kästner, Fabian Benduhn, Jens Meinicke, Gunter Saake, and Thomas Leich. 2014. FeatureIDE: An extensible framework for feature-oriented software development. Science of Computer Programming 79 (2014), 70--85.

Digital Library

[15]

Marc Thurley. 2006. sharpSAT-counting models with advanced component caching and implicit BCP. In International Conference on Theory and Applications of Satisfiability Testing. Springer, Springer Berlin Heidelberg, Berlin, Heidelberg, 424--429.

Digital Library

[16]

Tobias Pett, Thomas Thüm, Tobias Runge, Sebastian Krieter, Malte Lochau, and Ina Schaefer. 2019. Product Sampling for Product Lines: The Scalability Challenge. In Software Product Line Conference, Challenge Case. ACM, ACM, New York, NY, USA.

Digital Library

Cited By

Sundermann CLoth JThüm TFilkov VRay BZhou M(2024)Efficient Slicing of Feature Models via Projected d-DNNF CompilationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695594(1332-1344)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695594
Luo CLyu SZhao QWu WZhang HHu CChristakis MPradel M(2024)Beyond Pairwise Testing: Advancing 3-wise Combinatorial Interaction Testing for Highly Configurable SystemsProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680309(641-653)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680309
Pett TKrieter SThüm TSchaefer I(2024)MulTi-Wise Sampling: Trading Uniform T-Wise Feature Interaction Coverage for Smaller SamplesProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3672589(47-53)Online publication date: 2-Sep-2024
https://dl.acm.org/doi/10.1145/3646548.3672589
Show More Cited By

Index Terms

t-wise Coverage by Uniform Sampling
1. Software and its engineering
  1. Software creation and management
    1. Software development techniques
      1. Reusability
        Software product lines
2. Theory of computation
  1. Logic
    1. Automated reasoning

Recommendations

Baital: an adaptive weighted sampling approach for improved t-wise coverage
ESEC/FSE 2020: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

The rise of highly configurable complex software and its widespread usage requires design of efficient testing methodology. t-wise coverage is a leading metric to measure the quality of the testing suite and the underlying test generation engine. While ...
Product Sampling for Product Lines: The Scalability Challenge
SPLC '19: Proceedings of the 23rd International Systems and Software Product Line Conference - Volume A

Quality assurance for product lines is often infeasible for each product separately. Instead, only a subset of all products (i.e., a sample) is considered during testing such that at least the coverage of certain feature interactions is guaranteed. ...
MulTi-Wise Sampling: Trading Uniform T-Wise Feature Interaction Coverage for Smaller Samples
SPLC '24: Proceedings of the 28th ACM International Systems and Software Product Line Conference

Ensuring the functional safety of highly configurable systems often requires testing representative subsets of all possible configurations. The ratio of covered t-wise feature interactions is a common criterion for determining whether a subset of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

SPLC '19: Proceedings of the 23rd International Systems and Software Product Line Conference - Volume A

September 2019

356 pages

ISBN:9781450371384

DOI:10.1145/3336294

Editors:
Thorsten Berger
Chalmers | University of Gothenburg, Sweden
,
Philippe Collet
Université Côte d'Azur, France
,
Laurence Duchien
University Lille, France
,
Thomas Fogdal
Danfoss Power Electronics A/S, Denmark
,
Patrick Heymans
University of Namur, Belgium
,
Timo Kehrer
Humboldt-Universität zu Berlin, Germany
,
Jabier Martinez
Tecnalia, Spain
,
Raúl Mazo
University Paris 1 Panthéon-Sorbonne, France
,
Leticia Montalvillo
IK4-IKERLAN Research Center, Spain
,
Camille Salinesi
University Paris 1 Panthéon-Sorbonne, France
,
Xhevahire Tërnava
Sorbonne University, France
,
Thomas Thüm
TU Braunschweig, Germany
,
Tewfik Ziadi
Sorbonne University, France

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 September 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Artifacts Evaluated & Reusable

Author Tags

Qualifiers

Short-paper

Funding Sources

National Science Foundation

Conference

SPLC 2019

SPLC 2019: 23rd International Systems and Software Product Line Conference

September 9 - 13, 2019

Paris, France

Acceptance Rates

Overall Acceptance Rate 167 of 463 submissions, 36%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

26
Total Citations
View Citations
639
Total Downloads

Downloads (Last 12 months)172
Downloads (Last 6 weeks)28

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Sundermann CLoth JThüm TFilkov VRay BZhou M(2024)Efficient Slicing of Feature Models via Projected d-DNNF CompilationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695594(1332-1344)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695594
Luo CLyu SZhao QWu WZhang HHu CChristakis MPradel M(2024)Beyond Pairwise Testing: Advancing 3-wise Combinatorial Interaction Testing for Highly Configurable SystemsProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3650212.3680309(641-653)Online publication date: 11-Sep-2024
https://dl.acm.org/doi/10.1145/3650212.3680309
Pett TKrieter SThüm TSchaefer I(2024)MulTi-Wise Sampling: Trading Uniform T-Wise Feature Interaction Coverage for Smaller SamplesProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3672589(47-53)Online publication date: 2-Sep-2024
https://dl.acm.org/doi/10.1145/3646548.3672589
Burgstaller TGarber DLe VFelfernig A(2024)Optimization Space Learning: A Lightweight, Noniterative Technique for Compiler AutotuningProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3672588(36-46)Online publication date: 2-Sep-2024
https://dl.acm.org/doi/10.1145/3646548.3672588
Fernandez-Amoros DHeradio RHorcas Aguilera JGalindo JBenavides DFuentes L(2024)Pragmatic Random Sampling of the Linux Kernel: Enhancing the Randomness and Correctness of the conf ToolProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3672586(24-35)Online publication date: 2-Sep-2024
https://dl.acm.org/doi/10.1145/3646548.3672586
Yıldıran NOh JLawall JGazzillo P(2024)Maximizing Patch Coverage for Testing of Highly-Configurable Software without Exploding Build TimesProceedings of the ACM on Software Engineering10.1145/36437461:FSE(427-449)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3643746
Heß TSchmidt TOstheimer LKrieter SThüm T(2024)UnWise: High T-Wise Coverage from Uniform SamplingProceedings of the 18th International Working Conference on Variability Modelling of Software-Intensive Systems10.1145/3634713.3634716(37-45)Online publication date: 7-Feb-2024
https://dl.acm.org/doi/10.1145/3634713.3634716
Baranov EChakraborty SLegay AMeel KVinodchandran N(2024)A Scalable t-Wise Coverage Estimator: Algorithms and ApplicationsIEEE Transactions on Software Engineering10.1109/TSE.2024.341991950:8(2021-2039)Online publication date: Aug-2024
https://doi.org/10.1109/TSE.2024.3419919
Baranov ELegay A(2024)Baital: Sampling Configurable Systems with high t-wise coverageScience of Computer Programming10.1016/j.scico.2024.103209(103209)Online publication date: Sep-2024
https://doi.org/10.1016/j.scico.2024.103209
Entekhabi SMostowski WMousavi M(2023)Automated and Efficient Test-Generation for Grid-Based Multiagent Systems: Comparing Random Input Filtering versus Constraint SolvingACM Transactions on Software Engineering and Methodology10.1145/362473633:1(1-32)Online publication date: 23-Nov-2023
https://dl.acm.org/doi/10.1145/3624736
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten