article

Effective product-line testing using similarity-based product prioritization

Authors:

Mustafa Al-Hajjaji,

Gunter SaakeAuthors Info & Claims

Software and Systems Modeling (SoSyM), Volume 18, Issue 1

Pages 499 - 521

https://doi.org/10.1007/s10270-016-0569-2

Published: 01 February 2019 Publication History

Abstract

A software product line comprises a family of software products that share a common set of features. Testing an entire product-line product-by-product is infeasible due to the potentially exponential number of products in the number of features. Accordingly, several sampling approaches have been proposed to select a presumably minimal, yet sufficient number of products to be tested. Since the time budget for testing is limited or even a priori unknown, the order in which products are tested is crucial for effective product-line testing. Prioritizing products is required to increase the probability of detecting faults faster. In this article, we propose similarity-based prioritization, which can be efficiently applied on product samples. In our approach, we incrementally select the most diverse product in terms of features to be tested next in order to increase feature interaction coverage as fast as possible during product-by-product testing. We evaluate the gain in the effectiveness of similarity-based prioritization on three product lines with real faults. Furthermore, we compare similarity-based prioritization to random orders, an interaction-based approach, and the default orders produced by existing sampling algorithms considering feature models of various sizes. The results show that our approach potentially increases effectiveness in terms of fault detection ratio concerning faults within real-world product-line implementations as well as synthetically seeded faults. Moreover, we show that the default orders of recent sampling algorithms already show promising results, which, however, can still be improved in many cases using similarity-based prioritization.

References

[1]

Abal, I., Brabrand, C., Wasowski, A.: 42 variability bugs in the linux kernel: a qualitative analysis. In: Proc. Int'l Conf. Automated Software Engineering (ASE), pp. 421---432. ACM (2014)

Digital Library

[2]

Al-Hajjaji, M.: Scalable sampling and prioritization for product-line testing. In: Proc. Software Engineering (SE), pp. 295---298. Gesellschaft für Informatik (GI) (2015)

[3]

Al-Hajjaji, M., Thüm, T., Meinicke, J., Lochau, M., Saake, G.: Similarity-based prioritization in software product-line testing. In: Proc. Int'l Software Product Line Conf. (SPLC), pp. 197---206. ACM (2014)

Digital Library

[4]

Al-Hajjaji, M., Krieter, S., Thüm, T., Lochau, M., Saake, G.: IncLing: efficient product-line testing using incremental pairwise sampling. In: Proc. Int'l Conf. Generative Programming and Component Engineering (GPCE), pp. 144---155. ACM (2016)

Digital Library

[5]

Al-Hajjaji, M., Meinicke, J., Krieter, S., Schröter, R., Thüm, T., Leich, T., Saake, G.: Tool demo: testing configurable systems with featureIDE. In: Proc. Int'l Conf. Generative Programming and Component Engineering (GPCE), pp. 173---177. ACM (2016)

Digital Library

[6]

Apel, S., Batory, D., Kästner, C., Saake, G.: Feature-Oriented Software Product Lines: Concepts and Implementation. Springer, Berlin (2013)

Digital Library

[7]

Apel, S., von Rhein, A., Wendler, P., Größlinger, A., Beyer, D.: Strategies for product-line verification: case studies and experiments. In: Proc. Int'l Conf. Software Engineering (ICSE), pp. 482---491. IEEE (2013)

Digital Library

[8]

Arcuri, A., Briand, L.: A practical guide for using statistical tests to assess randomized algorithms in software engineering. In: Proc. Int'l Conf. Software Engineering (ICSE), pp. 1---10. ACM (2011)

Digital Library

[9]

Bagheri, E., Ensan, F., Gasevic, D.: Grammar-based test generation for software product line feature models. In: Proc. Conf. Centre for Advanced Studies on Collaborative Research (CASCON), pp. 87---101. IBM Corp. (2012)

Digital Library

[10]

Baller, H., Lochau, M.: Towards incremental test suite optimization for software product lines. In: Proc. Int'l Workshop Feature-Oriented Software Development (FOSD), pp. 30---36. ACM (2014)

Digital Library

[11]

Baller, H., Lity, S., Lochau, M., Schaefer, I.: Multi-objective test suite optimization for incremental product family testing. In: Proc. Int'l Conf. Software Testing, Verification and Validation (ICST), pp. 303---312. IEEE (2014)

Digital Library

[12]

Bryce, R.C., Colbourn, C.J.: One-test-at-a-time heuristic search for interaction test suites. In: Proc. Int'l Conf. on Genetic and Evolutionary Computation (GECCO), pp. 1082---1089. ACM (2007)

Digital Library

[13]

Bryce, R.C., Memon, A.M.: Test suite prioritization by interaction coverage. In: Workshop on Domain Specific Approaches to Software Test Automation: Conjunction with the 6th ESEC/FSE Joint Meeting, DOSTA '07, pp. 1---7. ACM (2007)

Digital Library

[14]

Bürdek, J., Lochau, M., Bauregger, S., Holzer, A., von Rhein, A., Apel, S., Beyer, D.: Facilitating reuse in multi-goal test-suite generation for software product lines. In: Egyed, A., Schaefer, I. (eds.) Fundamental Approaches to Software Engineering, volume 9033 of Lecture Notes in Computer Science, pp. 84---99. Springer (2015)

[15]

Carmo Machado, I.D., McGregor, J.D., Cavalcanti, YaC, De Almeida, E.S.: On strategies for testing software product lines: a systematic literature review. J. Inf. Softw. Technol. (IST) 56(10), 1183---1199 (2014)

Digital Library

[16]

Cartaxo, E.G., Machado, P.D., Neto, F.G.O.: On the use of a similarity function for test case selection in the context of model-based testing. Softw. Test. Verif. Reliab. (STVR) 21(2), 75---100 (2011)

Digital Library

[17]

Chvatal, V.: A greedy heuristic for the set-covering problem. Math. Oper. Res. 4(3), 233---235 (1979)

Digital Library

[18]

Cohen, M.B., Dwyer, M.B., Shi, J.: Interaction testing of highly-configurable systems in the presence of constraints. In: Proc. Int'l Symposium in Software Testing and Analysis (ISSTA), pp. 129---139. ACM (2007)

Digital Library

[19]

Devroey, X., Perrouin, G., Cordy, M., Schobbens, P.-Y., Legay, A., Heymans, P.: Towards statistical prioritization for software product lines testing. In: Proc. Int'l Workshop Variability Modelling of Software-Intensive Systems (VaMoS), pp. 10:1---10:7. ACM (2014)

Digital Library

[20]

Devroey, X., Perrouin, G., Legay, A., Schobbens, P.-Y., Heymans, P.: Search-based similarity-driven behavioural SPL testing. In: Proc. Int'l Workshop Variability Modelling of Software-Intensive Systems (VaMoS), pp. 89---96. ACM (2016)

Digital Library

[21]

Dietrich, C., Tartler, R., Schröder-Preikschat, W., Lohmann, D.: Understanding linux feature distribution. In: Proc. of Workshop on Modularity in Systems Software (MISS), pp. 15---20. ACM (2012)

Digital Library

[22]

Elbaum, S., Malishevsky, A.G., Rothermel, G.: Prioritizing test cases for regression testing. SIGSOFT Softw. Eng. Notes 25(5), 102---112 (2000)

Digital Library

[23]

Engström, E., Runeson, P.: Software product line testing--a systematic mapping study. J. Inf. Softw. Technol. (IST) 53(1), 2---13 (2011)

Digital Library

[24]

Ensan, A., Bagheri, E., Asadi, M., Gasevic, D., Biletskiy, Y.: Goal-oriented test case selection and prioritization for product line feature models. In: Proc. Int'l Conf. on Information Technology: New Generations (ITNG), pp. 291---298. IEEE (2011)

Digital Library

[25]

Ensan, F., Bagheri, E., Gasevic, D.: Evolutionary search-based test generation for software product line feature models. In: Proc. Int'l Conf. Advanced Information Systems Engineering (CAiSE), vol. 7328, pp. 613---628. Springer (2012)

Digital Library

[26]

Garvin, B.J., Cohen, M.B., Dwyer, M.B.: Evaluating improvements to a meta-heuristic search for constrained interaction testing. Empir. Softw. Eng. (EMSE) 16(1), 61---102 (2011)

Digital Library

[27]

Hall, R.J.: Fundamental nonmodularity in electronic mail. Autom. Softw. Eng. 12(1), 41---79 (2005)

Digital Library

[28]

Hamming, R.W.: Error detecting and error correcting codes. Bell Syst. Tech. J. 29(2), 147---160 (1950)

[29]

Harrold, M.J.: Testing: a roadmap. In: Proc. of the Conf. on The Future of Software Engineering (FSE), pp. 61---72. ACM (2000)

Digital Library

[30]

Hemmati, H., Briand, L.: An industrial investigation of similarity measures for model-based test case selection. In: Proc. Int'l Symposium Software Reliability Engineering (ISSRE), pp. 141---150. IEEE (2010)

Digital Library

[31]

Henard, C., Papadakis, M., Perrouin, G., Klein, J., Traon, Y.L.: PLEDGE: a product line editor and test generation tool. In: Proc. Int'l Software Product Line Conf. (SPLC), pp. 126---129. ACM (2013)

Digital Library

[32]

Henard, C., Papadakis, M., Perrouin, G., Klein, J., Heymans, P., Le Traon, Y.L.: Bypassing the combinatorial explosion: using similarity to generate and prioritize T-wise test configurations for software product lines. IEEE Trans. Softw. Eng. (TSE) 40(7), 650---670 (2014)

Digital Library

[33]

Jackson, M., Zave, P.: Distributed feature composition: a virtual architecture for telecommunications services. IEEE Trans. Softw. Eng. (TSE) 24(10), 831---847 (1998)

Digital Library

[34]

Johansen, M.F., Haugen, Ø., Fleurey, F.: Properties of realistic feature models make combinatorial testing of product lines feasible. In: Proc. Int'l Conf. Model Driven Engineering Languages and Systems (MODELS), pp. 638---652. Springer (2011)

Digital Library

[35]

Johansen, M.F., Haugen, Ø., Fleurey, F.: An algorithm for generating T-wise covering arrays from large feature models. In: Proc. Int'l Software Product Line Conf. (SPLC), pp. 46---55. ACM (2012)

Digital Library

[36]

Johansen, M.F., Haugen, Ø., Fleurey, F., Eldegard, A.G., Syversen, T.: Generating better partial covering arrays by modeling weights on sub-product lines. In: Proc. Int'l Conf. Model Driven Engineering Languages and Systems (MODELS), pp. 269---284. Springer (2012)

Digital Library

[37]

Kang, K.C., Cohen, S.G., Hess, J.A., Novak, W.E., Peterson, A.S.: Feature-oriented domain analysis (FODA) feasibility study. Technical Report CMU/SEI-90-TR-21, Software Engineering Institute (1990)

[38]

Kästner, C., Apel, S., ur Rahman, S.S., Rosenmüller, M., Batory, D., Saake, G.: On the impact of the optional feature problem: analysis and case studies. In: Proc. Int'l Software Product Line Conf. (SPLC), pp. 181---190. Software Engineering Institute (2009)

Digital Library

[39]

Kramer, J., Magee, J., Sloman, M., Lister, A.: CONIC: an integrated approach to distributed computer control systems. IEE Proc. Comput. Dig. Tech. 130(1), 1 (1983)

[40]

Kuhn, D.R., Wallace, D.R., Gallo Jr., A.M.: Software fault interactions and implications for software testing. IEEE Trans. Softw. Eng. (TSE) 30(6), 418---421 (2004)

Digital Library

[41]

Lachmann, R., Lity, S., Lischke, S., Beddig, S., Schulze, S., Schaefer, I.: Delta-oriented test case prioritization for integration testing of software product lines. In: Proc. Int'l Software Product Line Conf. (SPLC), pp. 81---90. ACM (2015)

Digital Library

[42]

Lei, Y., Kacker, R.N., Kuhn, D.R., Okun, V., Lawrence, J.: IPOG: a general strategy for T-way software testing. In: Proc. Int'l Conf. Engineering of Computer-Based Systems (ECBS), pp. 549---556. IEEE (2007)

Digital Library

[43]

McGregor, J.: Testing a software product line. In: Testing Techniques in Software Engineering, pp. 104---140. Springer, Berlin (2010)

[44]

Meinicke, J., Wong, C.-P., Kästner, C., Thüm, T., Saake, G.: On essential configuration complexity: measuring interactions in highly-configurable systems. In: Proc. Int'l Conf. Automated Software Engineering (ASE), pp. 483---494. ACM (2016)

Digital Library

[45]

Mendonça, M., Branco, M., Cowan, D.: SPLOT: software product lines online tools. In: Proc. Conf. Object-Oriented Programming, Systems, Languages and Applications (OOPSLA), pp. 761---762. ACM (2009)

Digital Library

[46]

Myers, G.J., Sandler, C., Badgett, T.: The Art of Software Testing, 3rd edn. Wiley, New York (2011)

Digital Library

[47]

Nguyen, H.V., Kästner, C., Nguyen, T.N.: Exploring variability-aware execution for testing plugin-based web applications. In: Proc. Int'l Conf. Software Engineering (ICSE), pp. 907---918. ACM (2014)

Digital Library

[48]

Northrop, L., Clements, P.: A framework for software product line practice, version 5.0. SEI (2007)

[49]

Oster, S., Markert, F., Ritter, P.: Automated incremental pairwise testing of software product lines. In: Proc. Int'l Software Product Line Conf. (SPLC), pp. 196---210. Springer (2010)

Digital Library

[50]

Perrouin, G., Sen, S., Klein, J., Baudry, B., Le Traon, Y.: Automated and scalable T-wise test case generation strategies for software product lines. In: Proc. Int'l Conf. Software Testing, Verification and Validation (ICST), pp. 459---468. IEEE (2010)

Digital Library

[51]

Perrouin, G., Oster, S., Sen, S., Klein, J., Baudry, B., Le Traon, Y.: Pairwise testing for software product lines: comparison of two approaches. Softw. Qual. J. (SQJ) 20(3---4), 605---643 (2012)

Digital Library

[52]

Plath, M., Ryan, M.: Feature integration using a feature construct. Sci. Comput. Program. (SCP) 41(1), 53---84 (2001)

Digital Library

[53]

Rothermel, G., Untch, R., Chu, C., Harrold, M.: Prioritizing test cases for regression testing. IEEE Trans. Softw. Eng. (TSE) 27(10), 929---948 (2001)

Digital Library

[54]

Sánchez, A.B., Segura, S., Ruiz-Cortés, A.: A comparison of test case prioritization criteria for software product lines. In: Proc. Int'l Conf. Software Testing, Verification and Validation (ICST), pp. 41---50. IEEE (2014)

Digital Library

[55]

Sánchez, A., Segura, S., Parejo, J., Ruiz-Cortés, A.: Variability testing in the wild: the drupal case study. Softw. Syst. Model. 1---22 (2015)

[56]

Shi, J., Cohen, M.B., Dwyer, M.B.: Integration testing of software product lines using compositional symbolic execution. In: Proc. Int'l Conf. Fundamental Approaches to Software Engineering (FASE), pp. 270---284. Springer (2012)

Digital Library

[57]

Thüm, T., Kästner, C., Erdweg, S., Siegmund, N.: Abstract features in feature modeling. In: Proc. Int'l Software Product Line Conf. (SPLC), pp. 191---200. IEEE (2011)

Digital Library

[58]

Thüm, T., Apel, S., Kästner, C., Schaefer, I., Saake, G.: A classification and survey of analysis strategies for software product lines. ACM Comput. Surv. 47(1), 6:1---6:45 (2014)

Digital Library

[59]

Thüm, T., Kästner, C., Benduhn, F., Meinicke, J., Saake, G., Leich, T.: FeatureIDE: an extensible framework for feature-oriented software development. Sci. Comput. Program. (SCP) 79, 70---85 (2014)

Digital Library

[60]

Utting, M., Legeard, B.: Practical Model-Based Testing: A Tools Approach. Morgan Kaufmann Publishers Inc., Burlington (2007)

Digital Library

[61]

Visser, W., Havelund, K., Brat, G.P., Park, S.: Model checking programs. In: Proc. Int'l Conf. Automated Software Engineering (ASE), pp. 3---12. Springer (2000)

Digital Library

[62]

Walcott, K.R., Soffa, M.L., Kapfhammer, G.M., Roos, R.S.: Time-aware test suite prioritization. In: Proc. Int'l Symposium in Software Testing and Analysis (ISSTA), pp. 1---12. ACM (2006)

Digital Library

[63]

Weiss, D.M.: The product line hall of fame. In: Proc. Int'l Software Product Line Conf. (SPLC), p. 395. IEEE (2008)

Digital Library

[64]

Yoo, S., Harman, N.: Regression testing minimization, selection and prioritization: a survey. Softw. Test. Verif. Reliab. (STVR) 22(2), 67---120 (2012)

Digital Library

[65]

Yoo, S., Harman, M., Tonella, P., Susi, A.: Clustering test cases to achieve effective and scalable prioritisation incorporating expert knowledge. In: Proc. Int'l Symposium in Software Testing and Analysis (ISSTA), ISSTA'09, pp. 201---212. ACM (2009)

Digital Library

Cited By

Medeiros RDíaz OAl-Hajjaji MSchulze M(2025)Portfolio Evolution Analysis for SPL ScopingJournal of Systems and Software10.1016/j.jss.2024.112259220:COnline publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1016/j.jss.2024.112259
Luo CSong JZhao QSun BChen JZhang HLin JHu C(2024)Solving the t-Wise Coverage Maximum Problem via Effective and Efficient Local Search-Based SamplingACM Transactions on Software Engineering and Methodology10.1145/368883634:1(1-64)Online publication date: 16-Aug-2024
https://dl.acm.org/doi/10.1145/3688836
Lyu JLi SZhang HYang LLiu BRigger M(2024)Towards Efficient Build Ordering for Incremental Builds with Multiple ConfigurationsProceedings of the ACM on Software Engineering10.1145/36607741:FSE(1494-1517)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660774
Show More Cited By

Effective product-line testing using similarity-based product prioritization
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis

Recommendations

Similarity-based prioritization in software product-line testing
SPLC '14: Proceedings of the 18th International Software Product Line Conference - Volume 1

Exhaustively testing every product of a software product line (SPL) is a difficult task due to the combinatorial explosion of the number of products. Combinatorial interaction testing is a technique to reduce the number of products under test. However, ...
IncLing: efficient product-line testing using incremental pairwise sampling
GPCE '16

A software product line comprises a family of software products that share a common set of features. It enables customers to compose software systems from a managed set of features. Testing every product of a product line individually is often ...
IncLing: efficient product-line testing using incremental pairwise sampling
GPCE 2016: Proceedings of the 2016 ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences

A software product line comprises a family of software products that share a common set of features. It enables customers to compose software systems from a managed set of features. Testing every product of a product line individually is often ...

Comments

Information & Contributors

Information

Published In

cover image Software and Systems Modeling (SoSyM)

Software and Systems Modeling (SoSyM) Volume 18, Issue 1

February 2019

773 pages

ISSN:1619-1366

Issue’s Table of Contents

Copyright © Copyright © 2019 Springer-Verlag GmbH Germany, part of Springer Nature.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 01 February 2019

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

27
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Medeiros RDíaz OAl-Hajjaji MSchulze M(2025)Portfolio Evolution Analysis for SPL ScopingJournal of Systems and Software10.1016/j.jss.2024.112259220:COnline publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1016/j.jss.2024.112259
Luo CSong JZhao QSun BChen JZhang HLin JHu C(2024)Solving the t-Wise Coverage Maximum Problem via Effective and Efficient Local Search-Based SamplingACM Transactions on Software Engineering and Methodology10.1145/368883634:1(1-64)Online publication date: 16-Aug-2024
https://dl.acm.org/doi/10.1145/3688836
Lyu JLi SZhang HYang LLiu BRigger M(2024)Towards Efficient Build Ordering for Incremental Builds with Multiple ConfigurationsProceedings of the ACM on Software Engineering10.1145/36607741:FSE(1494-1517)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660774
Sundermann CBrancaccio VKuiter EKrieter SHeß TThüm T(2024)Collecting Feature Models from the Literature: A Comprehensive Dataset for BenchmarkingProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3672590(54-65)Online publication date: 2-Sep-2024
https://dl.acm.org/doi/10.1145/3646548.3672590
Agh HAzamnouri AWagner S(2024)Software product line testing: a systematic literature reviewEmpirical Software Engineering10.1007/s10664-024-10516-x29:6Online publication date: 2-Sep-2024
https://dl.acm.org/doi/10.1007/s10664-024-10516-x
Xiang YHuang HLi SLi MLuo CYang X(2023)Automated Test Suite Generation for Software Product Lines Based on Quality-Diversity OptimizationACM Transactions on Software Engineering and Methodology10.1145/362815833:2(1-52)Online publication date: 22-Dec-2023
https://dl.acm.org/doi/10.1145/3628158
Medeiros RDíaz OBenavides DDe Roover CRumpe BShaikhha A(2023)Unleashing the Power of Implicit Feedback in Software Product Lines: Benefits AheadProceedings of the 22nd ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences10.1145/3624007.3624058(113-121)Online publication date: 22-Oct-2023
https://dl.acm.org/doi/10.1145/3624007.3624058
Bombarda ABonfanti SGargantini A(2023)On the Reuse of Existing Configurations for Testing Evolving Feature ModelsProceedings of the 27th ACM International Systems and Software Product Line Conference - Volume B10.1145/3579028.3609017(67-76)Online publication date: 28-Aug-2023
https://dl.acm.org/doi/10.1145/3579028.3609017
Luo CSong JZhao QLi YCai SHu C(2023)Generating Pairwise Covering Arrays for Highly Configurable Software SystemsProceedings of the 27th ACM International Systems and Software Product Line Conference - Volume A10.1145/3579027.3608998(261-267)Online publication date: 28-Aug-2023
https://dl.acm.org/doi/10.1145/3579027.3608998
Luo CZhao QCai SZhang HHu CRoychoudhury ACadar CKim M(2022)SamplingCA: effective and efficient sampling-based pairwise testing for highly configurable software systemsProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3549155(1185-1197)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3549155
Show More Cited By

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents