research-article

Sampling configurations from software product lines via probability-aware diversification and SAT solving

Authors:

Zhengxin Huang,

Miqing LiAuthors Info & Claims

Automated Software Engineering, Volume 29, Issue 2

https://doi.org/10.1007/s10515-022-00348-8

Published: 01 November 2022 Publication History

Abstract

Sampling a small, valid and representative set of configurations from software product lines (SPLs) is important, yet challenging due to a huge number of possible configurations to be explored. Recently, the sampling strategy based on satisfiability (SAT) solving has enjoyed great popularity due to its high efficiency and good scalability. However, this sampling offers no guarantees on diversity, especially in terms of the number of selected features, an important property to characterize a configuration. In this paper, we propose a probability-aware diversification (PaD) strategy to cooperate with SAT solving in generating diverse configurations, with the effect that valid configurations are efficiently generated by SAT solving while also maintaining diversity brought by PaD. Experimental results on 51 public SPLs show that, when working cooperatively with PaD, the performance (regarding diversity) of off-the-shelf SAT solvers has substantial improvements, with large effect sizes observed on more than 71% of all the cases. Furthermore, we propose a general search-based framework where PaD and evolutionary algorithms can work together, and instantiate this framework in the context of search-based diverse sampling and search-based multi-objective SPL configuration (where there is a practical need of generating diverse configurations). It is demonstrated by the experimental results that PaD also brings abundant performance gains to these search-based approaches. Finally, we apply PaD to a practical problem, i.e., machine learning based performance predictions of SPLs, and show that using PaD tends to improve the accuracy of performance prediction models.

References

[1]

Achlioptas D, Hammoudeh ZS, and Theodoropoulos P Beyersdorff O and Wintersteiger CM Fast sampling of perfectly uniform satisfying assignments Theory and applications of satisfiability testing - SAT 2018 2018 Cham Springer 135-147

[2]

Al-Hajjaji, M., Krieter, S., Thüm, T., et al: IncLing: Efficient Product-Line Testing Using Incremental Pairwise Sampling. In: Proceedings of the 2016 ACM SIGPLAN International Conference on generative programming: concepts and experiences. Association for Computing Machinery, New York, NY, USA, GPCE 2016, pp 144–155 (2016)

[3]

Alves Pereira, J., Acher, M., Martin, H., et al: Sampling effect on performance prediction of configurable systems: A case study. In: Proceedings of the ACM/SPEC International Conference on performance engineering. association for computing machinery, New York, NY, USA, ICPE’20, p 277-288, (2020)

[4]

Arcuri, A., Briand, L.: A practical guide for using statistical tests to assess randomized algorithms in software engineering. In: Proceedings of the 33rd International Conference on Software Engineering. Association for computing machinery, New York, NY, USA, ICSE’11, pp 1–10 (2011)

[5]

Arcuri A and Briand L A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering Software Test Verifi Reliab 2014 24 3 219-250

[6]

Balint, A., Schöning, U.: Choosing Probability Distributions for Stochastic Local Search and the Role of Make versus Break, International Conference on theory and applications of satisfiability testing, Berlin, Heidelberg, pp 16–29 (2012)

[7]

Baranov, E., Legay, A., Meel, K.S.: Baital: An adaptive weighted sampling approach for improved t-wise coverage. In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the foundations of software engineering. ACM, New York, NY, USA, ESEC/FSE 2020, pp 1114–1126 (2020)

[8]

Batory, D.: Feature models, grammars, and propositional formulas. In: Obbink H, Pohl K (eds) Proceedings of the 9th international conference software product lines, SPLC 2005. Springer Berlin Heidelberg, Berlin, Heidelberg, pp 7–20 (2005)

[9]

Berre DL and Parrain A The Sat4j library, release 2.2, system description J. Satisf. Bool. Model. Comput. 2010 7 59-64

[10]

Biere, A., Brummayer, R.: Picosat essentials. Journal on Satisfiability Boolean Modeling & Computation pp 75–97 (2008)

[11]

Cai, S.: Faster implementation for walksat. Tech. rep, Queensland Research Lab, NICTA, Australia (2013)

[12]

Cai S, Luo C, and Su K Improving WalkSAT By effective tie-breaking and efficient implementation Comput. J. 2014 58 11 2864-2875

[13]

Chakraborty S, Fremont DJ, Meel KS, et al. Baier C, Tinelli C, et al. On parallel scalable uniform SAT witness generation Tools Algorithms for the Construct. Anal. Syst. 2015 Berlin Heidelberg, Berlin, Heidelberg Springer 304-319

[14]

Chen J, Nair V, Krishna R, et al. ‘Sampling’ as a baseline optimizer for search-based software engineering IEEE Trans. Softw.Eng. 2019 45 6 597-614

[15]

Clements, P., Northrop, L.: Software product lines: practices and patterns. Addison-Wesley Longman Publishing Co., Inc (2001)

[16]

Coello Coello CA, Lamont GB, and Veldhuizen DAV Evolutionary algorithms for solving multi-objective problems 2007 2 LLC, New York, NY Springer

[17]

Cohen, M.B., Dwyer, M.B., Shi, J.: Interaction testing of highly-configurable systems in the presence of constraints. In: Proceedings of the 2007 International Symposium on software testing and analysis. association for computing machinery, New York, NY, USA, ISSTA’07, pp 129–139 (2007)

[18]

Cohen MB, Dwyer MB, and Shi J Constructing interaction test suites for highly-configurable systems in the presence of constraints: A greedy approach IEEE Trans. Softw. Eng. 2008 34 5 633-650

[19]

De Moura, L., Bjørner, N.: Z3: An efficient smt solver. In: Proceedings of the Theory and Practice of Software, 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems. Springer-Verlag, Berlin, Heidelberg, TACAS’08/ETAPS’08, pp 337–340 (2008)

[20]

Dutra R, Laeufer K, Bachrach J, et al. Efficient sampling of sat solutions for testing 2018 IEEE/ACM 40th international conference on software engineering (icse) 2018 Los Alamitos, CA, USA IEEE Computer Society 549-559

[21]

Garvin BJ, Cohen MB, and Dwyer MB Evaluating improvements to a meta-heuristic search for constrained interaction testing Empirical Softw. Eng. 2011 16 1 61-102

[22]

Gogate V and Dechter R Benhamou F A new algorithm for sampling csp solutions uniformly at random Principles and practice of constraint programming - CP 2006 2006 Berlin Heidelberg, Berlin, Heidelberg Springer 711-715

[23]

Guo J, White J, Wang G, et al. A genetic algorithm for optimized feature selection with resource constraints in software product lines Journal of Systems & Software 2011 84 12 2208-2221

[24]

Guo, J., Czarnecki, K., Apel, S., et al.: Variability-aware performance prediction: A statistical learning approach. In: 28th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp 301–311 (2013)

[25]

Guo, J., Zulkoski, E., Olaechea, R., et al: Scaling exact multi-objective combinatorial optimization by parallelization. In: ACM/IEEE International Conference on automated software engineering, ASE ’14, Vasteras, Sweden - September 15 - 19, 2014, pp 409–420 (2014)

[26]

Guo J, Liang JH, Shi K, et al. SMTIBEA: a hybrid multi-objective optimization algorithm for configuring large constrained software product lines Softw. Syst.Model. 2019 18 1447-1466

[27]

Halin A, Nuttinck A, Acher M, et al. Test them all, is it worth it? assessing configuration sampling on the jhipster web development stack Empirical Softw. Eng. 2019 24 674-717

[28]

Henard, C., Papadakis, M., Perrouin, G., et al: PLEDGE: A Product Line Editor and Test Generation Tool. In: Proceedings of the 17th International Software Product Line Conference Co-Located Workshops. Association for computing machinery, New York, NY, USA, SPLC’13 Workshops, pp 126–129 (2013)

[29]

Henard C, Papadakis M, Perrouin G, et al. Bypassing the combinatorial explosion: using similarity to generate and prioritize t-wise test configurations for software product lines IEEE Trans. Softw. Eng. 2014 40 7 650-670

[30]

Henard, C., Papadakis, M., Harman, M., et al: Combining multi-objective search and constraint solving for configuring large software product lines. In: The 37th International Conference on Software Engineering, pp 517–528 (2015)

[31]

Heradio, R., Fernandez-Amoros, D., Galindo, J.A., et al: Uniform and scalable sat-sampling for configurable systems. In: Proceedings of the 24th ACM Conference on Systems and Software Product Line: Volume A - Volume A. Association for computing machinery, New York, NY, USA, SPLC ’20 (2020)

[32]

Hierons RM, Li M, Liu X, et al. SIP: optimal product selection from feature models using many-objective evolutionary optimization ACM Trans. Softw. Eng. Methodol. 2016 25 2 17:1-17:39

[33]

Johansen, M.F., Haugen, Ø., Fleurey, F.: Properties of realistic feature models make combinatorial testing of product lines feasible. In: Proceedings of the 14th International Conference on model driven engineering languages and systems. Springer-Verlag, Berlin, Heidelberg, MODELS’11, pp 638–652 (2011)

[34]

Johansen, M.F., Haugen, Ø., Fleurey, F.: An algorithm for generating t-wise covering arrays from large feature models. In: Proceedings of the 16th International Software Product Line Conference - Volume 1. Association for computing machinery, New York, NY, USA, SPLC’12, pp 46–55 (2012)

[35]

Kaltenecker, C., Grebhahn, A., Siegmund, N., et al: Distance-based sampling of software configuration spaces. In: Proceedings of the 41st International Conference on software engineering. IEEE Press, ICSE’19, pp 1084–1094 (2019)

[36]

Kang, K.C., Cohen, S.G., Hess, J.A., et al: Feature-oriented domain analysis (FODA ) feasibility study. Tech. rep., CMU/SEI-90-TR-21. SEI. Georgetown University (1990)

[37]

Krieter, S., Thüm, T., Schulze, S., et al: YASA: yet another sampling algorithm. In: Cordy M, Acher M, Beuche D, et al (eds) VaMoS 20: 14th International Working Conference on variability modelling of software-intensive systems, Magdeburg Germany, February 5-7, 2020. ACM, pp 4:1–4:10 (2020)

[38]

Lehman J and Kenneth OS Abandoning objectives: evolution through the search for novelty alone Evolut. Comput. 2011 19 2 189-223

[39]

Lehman, J., Stanley, K.O.: Exploiting open-endedness to solve problems through the search for novelty. In: Proceedings of the Eleventh International Conference on Artificial Life (ALIFE XI), pp 329–336 (2008)

[40]

Li M and Yao X Quality evaluation of solution sets in multiobjective optimisation: a survey ACM Comput. Surv. 2019 52 2 26:1-26:38

[41]

Li M, Yang S, and Liu X Bi-goal evolution for many-objective optimization problems Artif. Intell. 2015 228 45-65

[42]

Li, M., Chen, T., Yao, X.: A critical review of: “a practical guide to select quality indicators for assessing pareto-based search algorithms in search-based software engineering”: Essay on quality indicator selection for sbse. In: Proceedings of the 40th International Conference on Software Engineering: New Ideas and Emerging Results. ACM, New York, NY, USA, ICSE-NIER ’18, pp 17–20, (2018)

[43]

Liang, J.H., Ganesh, V., Czarnecki, K., et al: SAT-based Analysis of Large Real-world Feature Models is Easy. In: Proceedings of the 19th International Conference on Software Product Line. ACM, New York, NY, USA, SPLC ’15, pp 91–100 (2015)

[44]

Liebig, J., von Rhein, A., Kästner, C., et al: Scalable analysis of variable software. In: Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. Association for Computing Machinery, New York, NY, USA, ESEC/FSE 2013, pp 81–91 (2013)

[45]

Lopez-Herrejon, R.E., Fischer, S., Ramler, R., et al: A first systematic mapping study on combinatorial interaction testing for software product lines. In: IEEE Eighth International Conference on Software testing, verification and validation workshops (ICSTW). IEEE, pp 1–10 (2015)

[46]

Luo, C., Sun, B., Qiao, B., et al: LS-Sampling: An effective local search based sampling approach for achieving high t-wise coverage. In: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Association for computing machinery, New York, NY, USA, ESEC/FSE 2021, pp 1081–1092 (2021)

[47]

Marques-Silva JP, Sakallah, et al. GRASP: a search algorithm for propositional satisfiability IEEE Trans. Comput. 1999 48 5 506-521

[48]

Medeiros, F., Kästner, C., Ribeiro, M., et al: A comparison of 10 sampling algorithms for configurable systems. In: 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE), pp 643–654 (2016)

[49]

Melo, J., Flesborg, E., Brabrand, C., et al: A quantitative analysis of variability warnings in linux. In: Proceedings of the Tenth International Workshop on variability modelling of software-intensive systems. ACM, New York, NY, USA, VaMoS’16, pp 3–8 (2016)

[50]

Mendonca, M., Branco, M., Cowan, D.: SPLOT: software product lines online tools. In: Proceedings of the 24th ACM SIGPLAN conference companion on Object oriented programming systems languages and applications, ACM, pp 761–762 (2009)

[51]

Oh, J., Batory, D., Myers, M., et al: Finding near-optimal configurations in product lines by random sampling. In: Proceedings of the 2017 11th Joint meeting on foundations of software engineering. ACM, New York, NY, USA, ESEC/FSE 2017, pp 61–71 (2017)

[52]

Oh, J., Gazzillo, P., Batory, D.: T-wise coverage by uniform sampling. In: Proceedings of the 23rd International Systems and Software Product Line Conference - Volume A. Association for computing machinery, New York, NY, USA, SPLC’19, pp 84–87 (2019)

[53]

Oh, J., Gazzillo, P., Batory, D., et al: Scalable uniform sampling for real-world software product lines. Tech. Rep. TR-20-01, The University of Texas at Austin (2020)

[54]

Olaechea, R., Rayside, D., Guo, J., et al: Comparison of exact and approximate multi-objective optimization for software product lines. In: The International software product line conference, pp 92–101 (2014)

[55]

Pett, T., Thüm, T., Runge, T., et al: Product sampling for product lines: The scalability challenge. In: Proceedings of the 23rd International Systems and Software Product Line Conference - Volume A. Association for Computing Machinery, New York, NY, USA, SPLC’19, pp 78–83 (2019)

[56]

Plazar, Q., Acher, M., Perrouin, G., et al: Uniform sampling of SAT solutions for configurable systems: Are we there yet? In: 12th IEEE Conference on software testing, validation and verification, ICST 2019, Xi’an, China, April 22-27, 2019. IEEE, pp 240–251 (2019)

[57]

Pohl, R., Lauenroth, K., Pohl, K.: A performance comparison of contemporary algorithmic approaches for automated analysis operations on feature models. In: IEEE/ACM International Conference on automated software engineering, pp 313–322 (2011)

[58]

Sarkar, A., Guo, J., Siegmund, N., et al: Cost-efficient sampling for performance prediction of configurable systems. In: Proceedings of the 30th IEEE/ACM International conference on automated software engineering. IEEE Press, ASE’15, pp 342–352 (2015)

[59]

Sayyad, A.S., Goseva-Popstojanova, K., Menzies, T., et al: On parameter tuning in search based software engineering: A replicated empirical study. In: International Workshop on Replication in empirical software engineering research, pp 84–90 (2013a)

[60]

Sayyad, A.S., Ingram, J., Menzies, T., et al: Optimum feature selection in software product lines:let your model and values guide your search. In: International Workshop on combining modelling and search-based software engineering, pp 22–27 (2013b)

[61]

Sayyad, A.S., Ingram, J., Menzies, T., et al: Scalable product line configuration: A straw to break the camel’s back. In: 2013 28th IEEE/ACM International conference on automated software engineering (ASE), pp 465–474 (2013c)

[62]

Sayyad, A.S., Menzies, T., Ammar, H.: On the value of user preferences in search-based software engineering: A case study in software product lines. In: 2013 35th International Conference on Software Engineering (ICSE), pp 492–501, (2013d)

[63]

Selman, B., Levesque, H., Mitchell, D.: A new method for solving hard satisfiability problems. In: Proceedings of the Tenth National Conference on artificial intelligence. AAAI Press, AAAI’92, pp 440–446 (1992)

[64]

Siegmund N, Grebhahn A, Apel S, et al. Performance-influence models for highly configurable systems Assoc. Comput. Mach. 2015 2015 284-294

[65]

Simon, L., Audemard, G.: Predicting Learnt Clauses Quality in Modern SAT Solver. In: Twenty-first International Joint Conference on artificial intelligence (IJCAI’09), Pasadena, United States (2009)

[66]

Soos, M., Gocht, S., Meel, K.S.: Tinted, detached, and lazy cnf-xor solving and its applications to counting and sampling. In: Proceedings of International Conference on Computer-Aided Verification (CAV) (2020)

[67]

Sundermann, C., Thüm, T., Schaefer, I.: Evaluating #sat solvers on industrial feature models. In: Proceedings of the 14th International Working Conference on Variability Modelling of Software-Intensive Systems. Association for Computing Machinery, New York, NY, USA, VAMOS’20, (2020)

[68]

Tan, T.H., Xue, Y., Chen, M., et al: Optimizing selection of competing features via feedback-directed evolutionary algorithms. In: Proceedings of the 2015 International Symposium on software testing and analysis (ISSTA 2015), pp 246–256 (2015)

[69]

Thurley M Biere A and Gomes CP Sharpsat - counting models with advanced component caching and implicit bcp Theor. Appl. Satisf. Test. 2006 Berlin Heidelberg, Berlin, Heidelberg Springer 424-429

[70]

Vargha A and Delaney HD A critique and improvement of the cl common language effect size statistics of mcgraw and wong J. Educ. Behav. Stat. 2000 25 2 101-132

[71]

Xiang Y, Zhou Y, Li M, et al. A vector angle based evolutionary algorithm for unconstrained many-objective problems IEEE Trans. Evol. Comput. 2017 21 1 131-152

[72]

Xiang Y, Zhou Y, Zheng Z, et al. Configuring software product lines by combining many-objective optimization and SAT solvers ACM Trans. Softw. Eng. Methodol. 2018 26 4 14:1-14:46

[73]

Xiang Y, Yang X, Zhou Y, et al. Going deeper with optimal software products selection using many-objective optimization and satisfiability solvers Empir. Softw. Eng. 2020 25 591-626

[74]

Xiang Y, Huang H, Li M, et al. Looking for novelty in search-based software product line testing IEEE Trans. Softw. Eng. 2022 48 7 2317-2338

[75]

Xiang, Y., Huang, H., Zhou, Y., et al: Search-based diverse sampling from real-world software product lines. In: Proceedings of the 44th International Conference on Software Engineering. ACM, New York, NY, USA, ICSE’22, pp 1945–1957, (2022)

[76]

Xue, Y., Li, Y.F.: Multi-objective Integer Programming Approaches for Solving Optimal Feature Selection Problem: A New Perspective on Multi-objective Optimization Problems in SBSE. In: Proceedings of the 40th International Conference on Software Engineering. ACM, New York, NY, USA, ICSE ’18, pp 1231–1242, (2018)

[77]

Xue Y, Li M, Shepperd M, et al. A novel aggregation-based dominance for pareto-based evolutionary algorithms to configure software product lines Neurocomputing 2019 364 32-48

[78]

Zhang, W., Sun, Z., Zhu, Q., et al: NLocalSAT: Boosting local search with solution prediction. In: Bessiere C (ed) Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020. ijcai.org, pp 1177–1183 (2020)

[79]

Zitzler, E., Künzli, S.: Indicator-based selection in multiobjective search. In: Proc. 8th International Conference on parallel problem solving from nature, PPSN VIII. Springer, pp 832–842 (2004)

[80]

Zitzler E and Thiele L Multiobjective evolutionary algorithms: a comparative case study and the strength pareto approach IEEE Trans. Evolut. Comput. 1999 3 4 257-271

Cited By

Huyghe MQuinton CRudametkin W(2024)Taming the Variability of Browser FingerprintsProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3672591(66-71)Online publication date: 2-Sep-2024
https://dl.acm.org/doi/10.1145/3646548.3672591
Knop UHofman PMihatsch MSiegmund M(2023)Balancing Variability and Costs in Software Product LinesProceedings of the 27th ACM International Systems and Software Product Line Conference - Volume A10.1145/3579027.3608992(213-222)Online publication date: 28-Aug-2023
https://dl.acm.org/doi/10.1145/3579027.3608992

Recommendations

Search-based diverse sampling from real-world software product lines
ICSE '22: Proceedings of the 44th International Conference on Software Engineering

Real-world software product lines (SPLs) often encompass enormous valid configurations that are impossible to enumerate. To understand properties of the space formed by all valid configurations, a feasible way is to select a small and valid sample set. ...
Product Sampling for Product Lines: The Scalability Challenge
SPLC '19: Proceedings of the 23rd International Systems and Software Product Line Conference - Volume A

Quality assurance for product lines is often infeasible for each product separately. Instead, only a subset of all products (i.e., a sample) is considered during testing such that at least the coverage of certain feature interactions is guaranteed. ...
Integrating Software Product Lines: A Study of Reuse versus Stability
COMPSAC '12: Proceedings of the 2012 IEEE 36th Annual Computer Software and Applications Conference

To achieve large-scale reuse and accelerate time-to-market, integration of multiple software product lines (SPLs) is becoming a trend. The integration of independently-developed SPLs enables the derivation of new products on demand in a particular ...

Comments

Information & Contributors

Information

Published In

cover image Automated Software Engineering

Automated Software Engineering Volume 29, Issue 2

Nov 2022

801 pages

ISSN:0928-8910

Issue’s Table of Contents

© The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2022. Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 November 2022

Accepted: 21 June 2022

Received: 03 December 2021

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China
Science and Technology Program of Guangzhou

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Huyghe MQuinton CRudametkin W(2024)Taming the Variability of Browser FingerprintsProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3672591(66-71)Online publication date: 2-Sep-2024
https://dl.acm.org/doi/10.1145/3646548.3672591
Knop UHofman PMihatsch MSiegmund M(2023)Balancing Variability and Costs in Software Product LinesProceedings of the 27th ACM International Systems and Software Product Line Conference - Volume A10.1145/3579027.3608992(213-222)Online publication date: 28-Aug-2023
https://dl.acm.org/doi/10.1145/3579027.3608992

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents