research-article

Public Access

Quality Indicators in Search-based Software Engineering: An Empirical Evaluation

Authors:

Dipesh Pradhan,

Safdar Aqeel Safdar,

Tao YueAuthors Info & Claims

ACM Transactions on Software Engineering and Methodology (TOSEM), Volume 29, Issue 2

Article No.: 10, Pages 1 - 29

https://doi.org/10.1145/3375636

Published: 04 March 2020 Publication History

All formats PDF

Abstract

Search-Based Software Engineering (SBSE) researchers who apply multi-objective search algorithms (MOSAs) often assess the quality of solutions produced by MOSAs with one or more quality indicators (QIs). However, SBSE lacks evidence providing insights on commonly used QIs, especially about agreements among them and their relations with SBSE problems and applied MOSAs. Such evidence about QIs agreements is essential to understand relationships among QIs, identify redundant QIs, and consequently devise guidelines for SBSE researchers to select appropriate QIs for their specific contexts. To this end, we conducted an extensive empirical evaluation to provide insights on commonly used QIs in the context of SBSE, by studying agreements among QIs with and without considering differences of SBSE problems and MOSAs. In addition, by defining a systematic process based on three common ways of comparing MOSAs in SBSE, we present additional observations that were automatically produced based on the results of our empirical evaluation. These observations can be used by SBSE researchers to gain a better understanding of the commonly used QIs in SBSE, in particular, regarding their agreements. Finally, based on the results, we also provide a set of guidelines for SBSE researchers to select appropriate QIs for their particular context.

References

[1]

Wasif Afzal, Richard Torkar, and Robert Feldt. 2009. A systematic review of search-based testing for non-functional system properties. Inf. Software Technol. 51, 6 (June 2009), 957--976.

Digital Library

[2]

Alan Agresti. 2013. Categorical Data Analysis. Wiley.

[3]

Shaukat Ali, Lionel C. Briand, Hadi Hemmati, and Rajwinder Kaur Panesar-Walawege. 2010. A systematic review of the application and empirical investigation of search-based test case generation. IEEE Trans. Software Eng. 36, 6 (Nov. 2010), 742--762.

Digital Library

[4]

Andrea Arcuri and Lionel Briand. 2011. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In Proceedings of the 33rd International Conference on Software Engineering (ICSE’11). ACM, New York, NY, 1--10.

Digital Library

[5]

Wesley Klewerton Guez Assunção, Thelma Elita Colanzi, Silvia Regina Vergilio, and Aurora Pozo. 2013. On the application of the multi-evolutionary and coupling-based approach with different aspect-class integration testing strategies. In Search Based Software Engineering, Günther Ruhe and Yuanyuan Zhang (Eds.). Springer Berlin, Germany, 19--33.

[6]

Márcio Barros and Arilo Neto. 2011. Threats to validity in search-based software engineering empirical studies. RelaTe-DIA 5 (01 2011).

[7]

Lionel Briand, Davide Falessi, Shiva Nejati, Mehrdad Sabetzadeh, and Tao Yue. 2012. Research-based innovation: A tale of three projects in model-driven engineering. In Proceedings of the International Conference on Model Driven Engineering Languages and Systems. Springer, 793--809.

Digital Library

[8]

Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20, 1 (1960), 37--46.

[9]

Altino Dantas, Italo Yeltsin, Allysson Allex Araújo, and Jerffeson Souza. 2015. Interactive software release planning with preferences base. In Proceedings of the International Symposium on Search Based Software Engineering. Springer, 341--346.

[10]

Kalyanmoy Deb and Deb Kalyanmoy. 2001. Multi-Objective Optimization Using Evolutionary Algorithms. John Wiley 8 Sons, Inc., New York, NY.

Digital Library

[11]

Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and T. A. M. T. Meyarivan. 2002. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6, 2 (2002), 182--197.

Digital Library

[12]

Juan J. Durillo and Antonio J. Nebro. 2011. jMetal: A Java framework for multi-objective optimization. Adv. Eng. Software 42, 10 (Oct. 2011), 760--771.

Digital Library

[13]

Juan J. Durillo, Antonio J. Nebro, Francisco Luna, and Enrique Alba. 2008. Solving three-objective optimization problems using a new hybrid cellular genetic algorithm. In Proceedings of the 10th International Conference on Parallel Problem Solving from Nature (PPSN X), Volume 5199. Springer-Verlag, New York, NY, 661--670.

[14]

Juan J. Durillo, YuanYuan Zhang, Enrique Alba, and Antonio J. Nebro. 2009. A study of the multi-objective next release problem. In Proceedings of the 2009 1st International Symposium on Search Based Software Engineering (SSBSE’09). IEEE Computer Society, Washington, DC, 49--58.

Digital Library

[15]

Carlos M. Fonseca and Peter J. Fleming. 1998. Multiobjective optimization and multiple constraint handling with evolutionary algorithms. I. A unified formulation. IEEE Trans. Syst. Man Cybern. Part A Syst. Humans 28, 1 (Jan. 1998), 26--37.

[16]

G. H. Freeman and J. H. Halton. 1951. Note on an exact treatment of contingency, goodness of fit and other problems of significance. Biometrika 38, 1–2 (1951), 141--149.

[17]

Giovani Guizzo, Silvia R. Vergilio, Aurora T. R. Pozo, and Gian M. Fritsche. 2017. A multi-objective and evolutionary hyper-heuristic applied to the integration and test order problem. Appl. Soft Comput. 56 (2017), 331--344.

Digital Library

[18]

Mark Harman and Bryan F. Jones. 2001. Search-based software engineering. Inf. Software Technol. 43, 14 (2001), 833--839.

[19]

Mark Harman, S. Afshin Mansouri, and Yuanyuan Zhang. 2012. Search-based software engineering: Trends, techniques and applications. ACM Comput. Surv. 45, 1, Article 11 (Dec. 2012), 1–61.

Digital Library

[20]

Joshua D. Knowles and David W. Corne. 2000. Approximating the nondominated front using the pareto archived evolution strategy. Evol. Comput. 8, 2 (June 2000), 149--172.

Digital Library

[21]

J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biom. 33, 1 (1977), 159–184.

[22]

Miqing Li, Tao Chen, and Xin Yao. 2018. A critical review of: “A practical guide to select quality indicators for assessing pareto-based search algorithms in search-based software engineering”: Essay on quality indicator selection for SBSE. In Proceedings of the 40th International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER’18). ACM, New York, NY, 17--20.

Digital Library

[23]

Miqing Li and Xin Yao. 2019. Quality evaluation of solution sets in multiobjective optimisation: A survey. Comput. Surv. 52, 2, Article 26 (March 2019), 38 pages.

[24]

R. E. Lopez-Herrejon, J. Ferrer, F. Chicano, A. Egyed, and E. Alba. 2014. Comparative analysis of classical multi-objective evolutionary algorithms and seeding strategies for pairwise testing of Software Product Lines. In Proceedings of the 2014 IEEE Congress on Evolutionary Computation (CEC). 387--396.

[25]

Roberto E. Lopez-Herrejon, Lukas Linsbauer, and Alexander Egyed. 2015. A systematic mapping study of search-based software engineering for software product lines. Inf. Software Technol. 61, C (May 2015), 33--51.

[26]

Francisco Luna, David L. González-Álvarez, Francisco Chicano, and Miguel A. Vega-Rodríguez. 2014. The software project scheduling problem: A scalability analysis of multi-objective metaheuristics. Appl. Soft Comput. 15 (2014), 136--148.

Digital Library

[27]

Phil McMinn. 2011. Search-based software testing: Past, present and future. In Proceedings of the 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops (ICSTW’11). IEEE Computer Society, Washington, DC, 153--163.

Digital Library

[28]

Antonio J. Nebro, Juan J. Durillo, Jose Garcia-Nieto, Carlos A. Coello Coello, Francisco Luna, and Enrique Alba. 2009. SMPSO: A new PSO-based metaheuristic for multi-objective optimization. In Proceedings of the 2009 IEEE Symposium on Computational Intelligence in Multi-Criteria Decision-Making (MCDM). 66--73.

[29]

Antonio J. Nebro, Juan J. Durillo, Francisco Luna, Bernabé Dorronsoro, and Enrique Alba. 2009. MOCell: A cellular genetic algorithm for multiobjective optimization. Int. J. Intell. Syst. 24, 7 (July 2009), 726--746.

[30]

Dipesh Pradhan, Shuai Wang, Shaukat Ali, and Tao Yue. 2016. Search-based cost-effective test case selection within a time budget: An empirical study. In Proceedings of the Genetic and Evolutionary Computation Conference 2016 (GECCO’16). ACM, New York, NY, 1085--1092.

Digital Library

[31]

Dipesh Pradhan, Shuai Wang, Shaukat Ali, Tao Yue, and Marius Liaaen. 2016. STIPI: Using search to prioritize test cases based on multi-objectives derived from industrial practice. In Proceedings of the IFIP International Conference on Testing Software and Systems. Springer, 172--190.

[32]

Dipesh Pradhan, Shuai Wang, Shaukat Ali, Tao Yue, and Marius Liaaen. 2018. REMAP: Using rule mining and multi-objective search for dynamic test case prioritization. In Proceedings of the IEEE 11th International Conference on Software Testing, Verification and Validation (ICST’18). IEEE, 46--57.

[33]

Aurora Ramírez, José Raúl Romero, and Christopher L. Simons. 2019. A systematic review of interaction in search-based software engineering. IEEE Trans. Software Eng. 45, 8 (Aug. 2019), 760--781.

[34]

Miha Ravber, Marjan Mernik, and Matej Črepinšek. 2017. The impact of quality indicators on the rating of multi-objective evolutionary algorithms. Appl. Soft Comput. 55, C (June 2017), 265--275.

[35]

Safdar Aqeel Safdar, Hong Lu, Tao Yue, and Shaukat Ali. 2017. Mining cross product line rules with multi-objective search and machine learning. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO’17). ACM, New York, NY, 1319--1326.

Digital Library

[36]

David J. Sheskin. 2011. Handbook of Parametric and Nonparametric Statistical Procedures (5 Ed.). Chapman 8 Hall/CRC.

[37]

Helge Spieker, Arnaud Gotlieb, Dusica Marijan, and Morten Mossige. 2017. Reinforcement learning for automatic test case prioritization and selection in continuous integration. In Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, 12--22.

Digital Library

[38]

M. Tanaka, H. Watanabe, Y. Furukawa, and T. Tanino. 1995. GA-based decision support system for multicriteria optimization. In 1995 IEEE International Conference on Systems, Man and Cybernetics. Intelligent Systems for the 21st Century, Vol. 2. 1556--1561.

[39]

David A. Van Veldhuizen and Gary B. Lamont. 2000. Multiobjective evolutionary algorithms: Analyzing the state-of-the-art. Evol. Comput. 8, 2 (June 2000), 125--147.

[40]

Niki Veček, Marjan Mernik, and Matej Črepinšek. 2014. A chess rating system for evolutionary algorithms: A new method for the comparison and ranking of evolutionary algorithms. Inf. Sci. 277 (2014), 656--679.

[41]

Shuai Wang, Shaukat Ali, and Arnaud Gotlieb. 2015. Cost-effective test suite minimization in product lines using search techniques. J. Syst. Softw. 103, C (May 2015), 370--391.

Digital Library

[42]

Shuai Wang, Shaukat Ali, Tao Yue, Yan Li, and Marius Liaaen. 2016. A practical guide to select quality indicators for assessing pareto-based search algorithms in search-based software engineering. In Proceedings of the 38th International Conference on Software Engineering (ICSE’16). ACM, New York, NY, 631--642.

Digital Library

[43]

Shuai Wang, Shaukat Ali, Tao Yue, and Marius Liaaen. 2015. UPMOA: An improved search algorithm to support user-preference multi-objective optimization. In Proceedings of the IEEE 26th International Symposium on Software Reliability Engineering (ISSRE’15). IEEE, 393--404.

Digital Library

[44]

Zai Wang, Ke Tang, and Xin Yao. 2008. A multi-objective approach to testing resource allocation in modular software systems. In Proceedings of the IEEE Congress on Evolutionary Computation (CEC’08), IEEE World Congress on Computational Intelligence, 2008). IEEE, 1148--1153.

[45]

Tao Yue and Shaukat Ali. 2014. Applying search algorithms for optimizing stakeholders familiarity and balancing workload in requirements assignment. In Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation (GECCO’14). ACM, New York, NY, 1295--1302.

Digital Library

[46]

Yuanyuan Zhang, Mark Harman, and Afshin Mansouri. 2017. The SBSE Repository: A repository and analysis of authors and research articles on Search Based Software Engineering. Retrieved October 22, 2019 from http://crestweb.cs.ucl.ac.uk/resources/sbse_repository/.

[47]

Aimin Zhou, Yaochu Jin, Qingfu Zhang, Bernhard Sendhoff, and Edward Tsang. 2006. Combining model-based and genetics-based offspring generation for multi-objective optimization using a convergence criterion. In Proceedings of the 2006 IEEE International Conference on Evolutionary Computation. 892--899.

[48]

E. Zitzler, M. Laumanns, and L. Thiele. 2001. SPEA2: Improving the strength Pareto evolutionary algorithm for multiobjective optimization. In Evolutionary Methods for Design Optimization and Control with Applications to Industrial Problems (19--21 Sept. 2001), K. C. Giannakoglou, D. T. Tsahalis, J. Périaux, K. D. Papailiou, and T. Fogarty (Eds.). International Center for Numerical Methods in Engineering, Athens, Greece, 95--100.

[49]

E. Zitzler and L. Thiele. 1999. Multiobjective evolutionary algorithms: A comparative case study and the strength Pareto approach. IEEE Trans. Evol. Comput. 3, 4 (Nov. 1999), 257--271.

Digital Library

Cited By

Zhong CLi SHuang HLiu XChen ZZhang YZhang H(2024)Domain-Driven Design for Microservices: An Evidence-Based InvestigationIEEE Transactions on Software Engineering10.1109/TSE.2024.338583550:6(1425-1449)Online publication date: Jun-2024
https://doi.org/10.1109/TSE.2024.3385835
Sun CLaurent TArcaini PIshikawa F(2024)Alternating Between Surrogate Model Construction and Search for Configurations of an Autonomous Delivery System2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER60148.2024.00045(383-394)Online publication date: 12-Mar-2024
https://doi.org/10.1109/SANER60148.2024.00045
Deshpande NMkaouer MOuni ASharma N(2024)Third-party software library migration at the method-level using multi-objective evolutionary searchSwarm and Evolutionary Computation10.1016/j.swevo.2023.10144484(101444)Online publication date: Mar-2024
https://doi.org/10.1016/j.swevo.2023.101444
Show More Cited By

Index Terms

Quality Indicators in Search-based Software Engineering: An Empirical Evaluation
1. Software and its engineering
  1. Software creation and management
    1. Search-based software engineering

Recommendations

A practical guide to select quality indicators for assessing pareto-based search algorithms in search-based software engineering
ICSE '16: Proceedings of the 38th International Conference on Software Engineering

Many software engineering problems are multi-objective in nature, which has been largely recognized by the Search-based Software Engineering (SBSE) community. In this regard, Pareto-based search algorithms, e.g., Non-dominated Sorting Genetic Algorithm ...
On the preferences of quality indicators for multi-objective search algorithms in search-based software engineering
Abstract
Multi-Objective Search Algorithms (MOSAs) have been applied to solve diverse Search-Based Software Engineering (SBSE) problems. In most cases, SBSE users select one or more commonly used MOSAs (for instance, Nondominated Sorting Genetic Algorithm ...
Do Quality Indicators Prefer Particular Multi-objective Search Algorithms in Search-Based Software Engineering?
Search-Based Software Engineering
Abstract
In Search-Based Software Engineering (SBSE), users typically select a set of Multi-Objective Search Algorithms (MOSAs) for their experiments without any justification, or they simply choose an MOSA because of its popularity (e.g., NSGA-II). On the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Software Engineering and Methodology

ACM Transactions on Software Engineering and Methodology Volume 29, Issue 2

April 2020

200 pages

ISSN:1049-331X

EISSN:1557-7392

DOI:10.1145/3386453

Editor:
Mauro Pezzè
Università della Svizzera italiana and Università di Milano-Bicocca, Switzerland

Issue’s Table of Contents

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 March 2020

Accepted: 01 December 2019

Revised: 01 December 2019

Received: 01 July 2019

Published in TOSEM Volume 29, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

National Natural Science Foundation of China
Norges Forskningsråd
Japan Science and Technology Agency

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

30
Total Citations
View Citations
1,214
Total Downloads

Downloads (Last 12 months)324
Downloads (Last 6 weeks)33

Reflects downloads up to 27 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhong CLi SHuang HLiu XChen ZZhang YZhang H(2024)Domain-Driven Design for Microservices: An Evidence-Based InvestigationIEEE Transactions on Software Engineering10.1109/TSE.2024.338583550:6(1425-1449)Online publication date: Jun-2024
https://doi.org/10.1109/TSE.2024.3385835
Sun CLaurent TArcaini PIshikawa F(2024)Alternating Between Surrogate Model Construction and Search for Configurations of an Autonomous Delivery System2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER60148.2024.00045(383-394)Online publication date: 12-Mar-2024
https://doi.org/10.1109/SANER60148.2024.00045
Deshpande NMkaouer MOuni ASharma N(2024)Third-party software library migration at the method-level using multi-objective evolutionary searchSwarm and Evolutionary Computation10.1016/j.swevo.2023.10144484(101444)Online publication date: Mar-2024
https://doi.org/10.1016/j.swevo.2023.101444
Venigalla AChimalakonda S(2024)An exploratory study of software artifacts on GitHub from the lens of documentationInformation and Software Technology10.1016/j.infsof.2024.107425(107425)Online publication date: Mar-2024
https://doi.org/10.1016/j.infsof.2024.107425
Yuste JPardo EDuarte AHao J(2024)Multi-objective general variable neighborhood search for software maintainability optimizationEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.108593133(108593)Online publication date: Jul-2024
https://doi.org/10.1016/j.engappai.2024.108593
Yu ZXu WSukjairungwattana PYu L(2023)A Meta-Analysis of Facebook-Assisted Learning Outcomes in Different Countries or RegionsInternational Journal of Information Technology and Web Engineering10.4018/IJITWE.31931218:1(1-18)Online publication date: 10-Mar-2023
https://dl.acm.org/doi/10.4018/IJITWE.319312
Formica FFan TMenghi C(2023)Search-Based Software Testing Driven by Automatically Generated and Manually Defined Fitness FunctionsACM Transactions on Software Engineering and Methodology10.1145/362474533:2(1-37)Online publication date: 23-Dec-2023
https://dl.acm.org/doi/10.1145/3624745
Lu CGrundy J(2023)Evolutionary Computation and Reinforcement Learning for Cyber-Physical System DesignProceedings of the 45th International Conference on Software Engineering: Companion Proceedings10.1109/ICSE-Companion58688.2023.00071(264-266)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE-Companion58688.2023.00071
Victorica MArcaini PIshikawa FKawamoto HSawai KMuramoto E(2023)Stability-aware Exploration of Design Space of Autonomous Robots for Goods Delivery2023 27th International Conference on Engineering of Complex Computer Systems (ICECCS)10.1109/ICECCS59891.2023.00030(177-186)Online publication date: 14-Jun-2023
https://doi.org/10.1109/ICECCS59891.2023.00030
Arcaini PCastellano EIshikawa FKawamoto HSawai KMuramoto E(2023)Incremental Search-Based Allocation of Autonomous Robots for Goods Delivery2023 IEEE Congress on Evolutionary Computation (CEC)10.1109/CEC53210.2023.10254103(1-10)Online publication date: 1-Jul-2023
https://doi.org/10.1109/CEC53210.2023.10254103
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents