research-article

Multiple objective test set selection for software product line testing: evaluating different preference-based algorithms

Authors:

Helson Luiz Jakubovski Filho,

Thiago Nascimento Ferreira,

Silvia Regina VergilioAuthors Info & Claims

SBES '18: Proceedings of the XXXII Brazilian Symposium on Software Engineering

Pages 162 - 171

https://doi.org/10.1145/3266237.3266275

Published: 17 September 2018 Publication History

Abstract

The selection of optimal test sets for Software Product Lines (SPLs) is a complex task impacted by many factors and that needs to consider the tester's preferences. To help in this task, Preference-based Evolutionary Multi-objective Algorithms (PEMOAs) have been explored. They use a Reference Point (RP), which represents the user preference and guides the search, resulting in a greater number of solutions in the ROI (Region of Interest). This region contains solutions that are more interesting from the tester's point of view. However, the explored PEMOAs have not been compared yet and the results reported in the literature do not consider many-objective formulations. Such an evaluation is important because in the presence of more than three objectives the performance of the algorithms may change and the number of solutions increases. Considering this fact, this work presents evaluation results of four PEMOAs for selection of products in the SPL testing considering cost, testing criteria coverage, products similarity, and the number of revealed faults, given by the mutation score. The PEMOAs present better performance than traditional algorithms, avoiding uninteresting solutions. We introduce a hyper-heuristic version of the PEMOA R-NSGA-II that presents the best results in a general case.

References

[1]

E. K. Burke, M. Gendreau, M. Hyde, G. Kendall, G. Ochoa, E. Özcan, and R. Qu. 2013. Hyper-heuristics: A survey of the state of the art. J. Oper. Res. Soc. 64, 12 (Dec. 2013), 1695.

[2]

D. M. Cohen, S. R. Dalal, J. Parelius, and G. C. Patton. 1996. The combinatorial design approach to automatic test generation. IEEE Software 13, 5 (1996), 83--88.

Digital Library

[3]

M. B. Cohen, M. B. Dwyer, and J. Shi. 2006. Coverage and Adequacy in Software Product Line Testing. In Proceedings of ROSATEA'06. ACM, Portland, USA, 53--63.

Digital Library

[4]

K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan. 2002. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6, 2 (2002), 182--197.

Digital Library

[5]

K. Deb and J. Sundar. 2006. Reference Point Based Multi-objective Optimization Using Evolutionary Algorithms. In Proceedings of GECCO'06. ACM, Seattle, USA, 635--642.

Digital Library

[6]

J. J. Durillo and A.J. Nebro. 2011. jMetal: A Java framework for multi-objective optimization. Adv. Eng. Softw. 42 (2011), 760--771.

Digital Library

[7]

J. M. Ferreira, S. R. Vergilio, and M. A. Quináia. 2017. A Mutation Approach to Feature Testing of Software Product Lines. Int. J. Softw. Eng. Know. 27, 5 (2017), 232--237.

[8]

T. N. Ferreira, J. A. Prado Lima, A. Strickler, J. N. Kuk, S. R. Vergilio, and A. Pozo. 2017. Hyper-heuristic Based Product Selection for Software Product Line Testing. IEEE Comp. Intell. Mag. 12, 2 (May 2017), 34--45.

Digital Library

[9]

T. N. Ferreira, S. R. Vergilio, and J. T. de Souza. 2017. Incorporating User Preferences in Search-Based Software Engineering: A Systematic Mapping Study. Info. Softw. Tech. 90 (2017), 55--69.

[10]

C. Henard, M. Papadakis, M. Harman, and Y. Le Traon. 2015. Combining Multi-Objective Search and Constraint Solving for Configuring Large Software Product Lines. In Proceedings of ICSE'15, Vol. 1. IEEE, Florence, Italy, 517--528.

Digital Library

[11]

C. Henard, M. Papadakis, G. Perrouin, J. Klein, P. Heymans, and Y. Le Traon. 2014. Bypassing the combinatorial explosion: Using similarity to generate and prioritize T-wise test configurations for software product lines. IEEE Trans. Softw. Eng. 40, 7 (2014), 650--670.

Digital Library

[12]

P. Jaccard. 1908. Nouvelles recherches sur la distribution florale. Bulletin de la Societe Vaudoise des Sciences Naturelles 44 (1908), 223--270.

[13]

H. L. Jakubovski-Filho, T. N. Ferreira, and S. R. Vergilio. 2018. Incorporating User Preferences in a Software Product Line Testing Hyper-Heuristic Approach. In Proceedings of CEC'18. IEEE, Rio de Janeiro, Brazil, 2283--2290.

[14]

H. L. Jakubovski-Filho, T. N. Ferreira, and S. R. Vergilio. 2018. Preference Based Multi-Objective Algorithms Applied to the Variability Testing of Software Product Lines. J. Syst. Softw. (2018). Under Review.

[15]

S. Kalboussi, S. Bechikh, M. Kessentini, and L. B. Said. 2013. Preference-based many-objective evolutionary testing generates harder test cases for autonomous agents. In Proceedings of SSBSE'13. Springer, St. Petersburg, Russia, 245--250.

Digital Library

[16]

K. C. Kang, J. Lee, and P. Donohoe. 2002. Feature-Oriented Project Line Engineering. IEEE Software 19, 4 (July 2002), 58--65.

Digital Library

[17]

C. H. P. Kim, D. S. Batory, and S. Khurshid. 2011. Reducing Combinatorics in Testing Product Lines. In Proceedings of AOSD'11. ACM, Porto de Galinhas, Brazil, 57--68.

Digital Library

[18]

C. H. P. Kim, D. Marinov, S. Khurshid, D. Batory, S. Souto, P. Barros, and M. D'Amorim. 2013. SPLat: Lightweight Dynamic Analysis for Reducing Combinatorics in Testing Configurable Systems. In Proceedings of FSE. ACM, Saint Petersburg, Russia, 257--267.

Digital Library

[19]

B. P. Lamancha and M. P. Usaola. 2010. Testing product generation in software product lines using pairwise for features coverage. In Proceedings of ICTSS'10. Springer, Natal, Brazil, 111--125.

Digital Library

[20]

K. Li, K. Deb, and X. Yao. 2017. R-Metric: Evaluating the Performance of Preference-Based Evolutionary Multi-Objective Optimization Using Reference Points. IEEE Trans. Evol. Comput. (2017), 1--15.

[21]

K. Li, A. Fialho, S. Kwong, and Q. Zhang. 2014 Adaptive Operator Selection With Bandits for a Multiobjective Evolutionary Algorithm Based on Decomposition. IEEE Trans. Evol. Comput. 18, 1 (Feb. 2014), 114--130.

[22]

M. Li, T. Chen, and X. Yao. 2018. A Critical Review of "A Practical Guide to Select Quality Indicators for Assessing Pareto-Based Search Algorithms in Search-Based Software Engineering": Essay on Quality Indicator Selection for SBSE. In Proceedings of ICSE'18. ACM, Gothenburg, Sweden.

Digital Library

[23]

R. E. Lopez-Herrejon, F. Chicano, J. Ferrer, A. Egyed, and E. Alba. 2013. Multi-objective optimal test suite computation for software product line pairwise testing. In Proceedings of ICSM'13. IEEE, Eindhoven, The Netherlands, 404--407.

Digital Library

[24]

B. Marculescu, R. Feldt, and R. Torkar. 2012. A concept for an interactive search-based software testing system. In Proceedings of SSBSE'12. Springer, Riva del Garda, Italy, 273--278.

Digital Library

[25]

B. Marculescu, R. Feldt, and R. Torkar. 2013. Objective re-weighting to guide an interactive search based software testing system. In Proceedings of ICMLA'13. IEEE, Miami, USA.

Digital Library

[26]

R. A. Matnei-Filho and S. R. Vergilio. 2016. A multi-objective test data generation approach for mutation testing of feature models. J. Softw. Eng. Res. Dev. 4, 1 (2016), 1--29.

[27]

S. Oster, M. Zink, M. Lochau, and M. Grechanik. 2011. Pairwise feature-interaction testing for SPLs: Potentials and limitations. In Proceedings of SPLC'11, Vol. 2. ACM, Munich, Germany, 1--8.

Digital Library

[28]

D. Reuling, J. Bürdek, S. Rotärmel, M. Lochau, and U. Kelter. 2015. Fault-based Product-line Testing: Effective Sample Generation Based on Feature-diagram Mutation. In Proceedings of SPLC'15. ACM, Nashville, USA, 131--140.

Digital Library

[29]

L. B. Said, S. Bechikh, and K Ghédira. 2010. The r-dominance: A new dominance relation for interactive evolutionary multicriteria decision making. IEEE Trans. Evol. Comput. 14, 5 (2010), 801--818.

Digital Library

[30]

A. S. Sayyad, T. Menzies, and H. Ammar. 2013. On the Value of User Preferences in Search-based Software Engineering: A Case Study in Software Product Lines. In Proceedings of ICSE'13. IEEE Press, San Francisco, USA, 492--501.

Digital Library

[31]

A. Strickler, J. A. Prado Lima, S. R. Vergilio, and A. T. R. Pozo. 2016. Deriving products for variability test of Feature Models with a hyper-heuristic approach. Appl. Softw. Comput. 49 (2016), 1232--1242.

Digital Library

[32]

H. Takagi. 2001. Interactive evolutionary computation: Fusion of the capabilities of EC optimization and human evaluation. Proceedings of IEEE'01 89, 9 (September 2001), 1275--1296.

Digital Library

[33]

P. Trinidad, D. Benavides, A. Ruiz-Cortés, S. Segura, and A. Jimenez. 2008. FaMa framework. In Proceedings of SPLC'08. IEEE, Limerick, Ireland, 359--359.

Digital Library

[34]

A. E. E. Yamany and M. S. Elgamel. 2015. Smart OptiSelect Preference Based Innovative Framework for User-in-the-Loop Feature Selection in Software Product Lines. In Proceedings of ICIT'15. ACM, Amman, Jordan, 657--666.

[35]

A. E. E. Yamany, M. Shaheen, and A. S. Sayyad. 2014. OPTI-SELECT: An interactive tool for user-in-the-loop feature selection in software product lines. In Proceedings of SPLC'14. ACM, Florence, Italy, 126--129.

Digital Library

[36]

E. Zitzler. 1999. Evolutionary Algorithms for Multiobjective Optimization: Methods and Applications. Ph.D. Dissertation. ETH Zurich, Switzerland.

Cited By

Almulla HGay G(2022)Learning how to search: generating effective test cases through adaptive fitness function selectionEmpirical Software Engineering10.1007/s10664-021-10048-827:2Online publication date: 11-Jan-2022
https://doi.org/10.1007/s10664-021-10048-8
Ferreira TVergilio SKessentini M(2021)Implementing Search-Based Software Engineering Approaches with NautilusProceedings of the XXXV Brazilian Symposium on Software Engineering10.1145/3474624.3476010(303-308)Online publication date: 27-Sep-2021
https://dl.acm.org/doi/10.1145/3474624.3476010
Balera JSantiago Júnior V(2019)A systematic mapping addressing Hyper-Heuristics within Search-based Software TestingInformation and Software Technology10.1016/j.infsof.2019.06.012114:C(176-189)Online publication date: 1-Oct-2019
https://dl.acm.org/doi/10.1016/j.infsof.2019.06.012

Index Terms

Multiple objective test set selection for software product line testing: evaluating different preference-based algorithms
1. Software and its engineering
  1. Software creation and management
    1. Search-based software engineering

Recommendations

Automatic Generation of Search-Based Algorithms Applied to the Feature Testing of Software Product Lines
SBES '17: Proceedings of the XXXI Brazilian Symposium on Software Engineering

The selection of products for the variability testing of Feature Models (FMs) is a complex task impacted by many factors. To solve this problem, Multi-Objective Evolutionary Algorithms (MOEAs) have been successfully used in the field known as Search-...
Preference based multi-objective algorithms applied to the variability testing of software product lines
Highlights
- Preference-Based Evolutionary Multi-objective Algorithms applied to SPL testing.
Abstract
Evolutionary Multi-Objective Algorithms (EMOAs) have been applied to derive products for the variability testing of Software Product Lines (SPLs), which is a complex task impacted by many factors, such as the number of products to be ...
Regression Test Selection and Product Line System Testing
ICST '10: Proceedings of the 2010 Third International Conference on Software Testing, Verification and Validation

Context: Software product lines (SPL) are used in industry to achieve more efficient software development. To test a SPL is complex and costly and often becomes a bottleneck in the product line organization. Objective: This research aims to develop and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

SBES '18: Proceedings of the XXXII Brazilian Symposium on Software Engineering

September 2018

379 pages

ISBN:9781450365031

DOI:10.1145/3266237

Program Chairs:
Uirá Kulesza
UFRN
,
Rafael Prikladnicki
PUCRS
,
Marco Aurélio Gerosa
Northern Arizona University
,
Claudia Werner
COPPE/UFRJ
,
Rossana Andrade
UFC

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 September 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SBES '18

SBES '18: XXXII BRAZILIAN SYMPOSIUM ON SOFTWARE ENGINEERING

September 17 - 21, 2018

Sao Carlos, Brazil

Acceptance Rates

SBES '18 Paper Acceptance Rate 38 of 140 submissions, 27%;

Overall Acceptance Rate 147 of 427 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
125
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)0

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Almulla HGay G(2022)Learning how to search: generating effective test cases through adaptive fitness function selectionEmpirical Software Engineering10.1007/s10664-021-10048-827:2Online publication date: 11-Jan-2022
https://doi.org/10.1007/s10664-021-10048-8
Ferreira TVergilio SKessentini M(2021)Implementing Search-Based Software Engineering Approaches with NautilusProceedings of the XXXV Brazilian Symposium on Software Engineering10.1145/3474624.3476010(303-308)Online publication date: 27-Sep-2021
https://dl.acm.org/doi/10.1145/3474624.3476010
Balera JSantiago Júnior V(2019)A systematic mapping addressing Hyper-Heuristics within Search-based Software TestingInformation and Software Technology10.1016/j.infsof.2019.06.012114:C(176-189)Online publication date: 1-Oct-2019
https://dl.acm.org/doi/10.1016/j.infsof.2019.06.012

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents