research-article

A critical review of: "a practical guide to select quality indicators for assessing pareto-based search algorithms in search-based software engineering": essay on quality indicator selection for SBSE

Authors:

Xin YaoAuthors Info & Claims

ICSE-NIER '18: Proceedings of the 40th International Conference on Software Engineering: New Ideas and Emerging Results

Pages 17 - 20

https://doi.org/10.1145/3183399.3183405

Published: 27 May 2018 Publication History

Abstract

This paper presents a critical review of the work published at ICSE'2016 on a practical guide of quality indicator selection for assessing multiobjective solution sets in search-based software engineering (SBSE). This review has two goals. First, we aim at explaining why we disagree with the work at ICSE'2016 and why the reasons behind this disagreement are important to the SBSE community. Second, we aim at providing a more clarified guide of quality indicator selection, serving as a new direction on this particular topic for the SBSE community. In particular, we argue that it does matter which quality indicator to select, whatever in the same quality category or across different categories. This claim is based upon the fundamental goal of multiobjective optimisation --- supplying the decision-maker a set of solutions which are the most consistent with their preferences.

References

[1]

B. Bozkurt, J. W. Fowler, E. S. Gel, B. Kim, M. Köksalan, and J. Wallenius. Quantitative comparison of approximate solution sets for multicriteria optimization problems with weighted Tchebycheff preference function. Operations Research, 58(3):650--659, 2010.

Digital Library

[2]

T. Chen and R. Bahsoon. Self-adaptive trade-off decision making for autoscaling cloud-based services. IEEE Transactions on Services Computing, 10(4):618--632, 2017.

[3]

J. L. Cochrane and M. Zeleny. Multiple Criteria Decision Making. University of South Carolina Press, 1973.

[4]

C. A. C. Coello and M. R. Sierra. A study of the parallelization of a revolutionary multi-objective evolutionary algorithm. In Proceedings of the Mexican International Conference on Artificial Intelligence (MICAI), pages 688--697, 2004.

[5]

C. Henard, M. Papdakis, M. Harman, and Y. L. Traon. Combining multi-objective search and constraint solving for configuring large software product lines. In Proceedings of the 2015 International Conference on Software Engineering, pages 517--528, 2015.

Digital Library

[6]

R. M. Hierons, M. Li, X. Liu, S. Segura, and W. Zheng. SIP: optimal product selection from feature models using many-objective evolutionary optimization. ACM Transactions on Software Engineering and Methodology, 25(2):17, 2016.

Digital Library

[7]

H. Ishibuchi, H. Masuda, Y. Tanigaki, and Y. Nojima. Modified distance calculation in generational distance and inverted generational distance. In Evolutionary Multi-Criterion Optimization (EMO), pages 110--125. 2015.

[8]

M. Li, S. Yang, and X. Liu. Diversity comparison of Pareto front approximations in many-objective optimization. IEEE Transactions on Cybernetics, 44(12):2568--2584, 2014.

[9]

M. Monperrus. A critical review of automatic patch generation learned from human-written patches: essay on the problem statement and the evaluation of automatic software repair. In Proceedings of the 36th International Conference on Software Engineering, pages 234--242, 2014.

Digital Library

[10]

T. Saber, D. Brevet, G. Botterweck, and A. Ventresque. Is seeding a good strategy in multi-objective feature selection when feature models evolve? Information and Software Technology, 2017.

[11]

T. Saber, J. Thorburn, L. Murphy, and A. Ventresque. VM reassignment in hybrid clouds for large decentralised companies: A multi-objective challenge. Future Generation Computer Systems, 2017.

Digital Library

[12]

S. Sayin. Measuring the quality of discrete representations of efficient sets in multiple objective mathematical programming. Mathematical Programming, 87(3):543--560, 2000.

[13]

D. A. Van Veldhuizen and G. B. Lamont. Evolutionary computation and convergence to a Pareto front. In Late Breaking Papers at the Genetic Programming Conference, pages 221--228, 1998.

[14]

H. Wada, J. Suzuki, Y. Yamano, and K. Oba. E3: A multiobjective optimization framework for sla-aware service composition. IEEE Transactions on Services Computing, 5(3):358--372, 2012.

Digital Library

[15]

S. Wang, S. Ali, T. Yue, Y. Li, and M. Liaaen. A practical guide to select quality indicators for assessing Pareto-based search algorithms in search-based software engineering. In Proceedings of the 38th International Conference on Software Engineering (ICSE), pages 631--642, 2016.

Digital Library

[16]

W. Zheng, R. M. Hierons, M. Li, X. Liu, and V. Vinciotti. Multi-objective optimisation for regression testing. Information Sciences, 334:1--16, 2016.

Digital Library

[17]

A. Zhou, Y. Jin, Q. Zhang, B. Sendhoff, and E. Tsang. Combining model-based and genetics-based offspring generation for multi-objective optimization using a convergence criterion. In IEEE Congress on Evolutionary Computation, pages 892--899, 2006.

[18]

E. Zitzler, D. Brockhoff, and L. Thiele. The hypervolume indicator revisited: On the design of Pareto-compliant indicators via weighted integration. In Evolutionary multi-criterion optimization, pages 862--876, 2007.

[19]

E. Zitzler and L. Thiele. Multiobjective evolutionary algorithms: A comparative case study and the strength Pareto approach. IEEE Transactions on Evolutionary Computation, 3(4):257--271, 1999.

Digital Library

[20]

E. Zitzler, L. Thiele, M. Laumanns, C. M. Fonseca, and V. G. Da Fonseca. Performance assessment of multiobjective optimizers: An analysis and review. IEEE Transactions on Evolutionary Computation, 7(2):117--132, 2003.

Digital Library

Cited By

Chen TLi M(2023)The Weights Can Be Harmful: Pareto Search versus Weighted Search in Multi-objective Search-based Software EngineeringACM Transactions on Software Engineering and Methodology10.1145/351423332:1(1-40)Online publication date: 13-Feb-2023
https://dl.acm.org/doi/10.1145/3514233
Li MChen T(2023)Methodology and Guidelines for Evaluating Multi-Objective Search-Based Software Engineering2023 IEEE/ACM 45th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)10.1109/ICSE-Companion58688.2023.00096(338-339)Online publication date: May-2023
https://doi.org/10.1109/ICSE-Companion58688.2023.00096
Pallathadka LPallathadka H(2022)Racism and Health: Evidence and Needed ResearchIntegrated Journal for Research in Arts and Humanities10.55544/ijrah.2.6.172:6(128-136)Online publication date: 25-Nov-2022
https://doi.org/10.55544/ijrah.2.6.17
Show More Cited By

Index Terms

A critical review of: "a practical guide to select quality indicators for assessing pareto-based search algorithms in search-based software engineering": essay on quality indicator selection for SBSE
1. Software and its engineering
  1. Software creation and management
    1. Search-based software engineering

Recommendations

On the Preferences of Quality Indicators for Multi-Objective Search Algorithms in Search-Based Software Engineering (Hot Off the Press track at GECCO 2023)
GECCO '23 Companion: Proceedings of the Companion Conference on Genetic and Evolutionary Computation

As a search-based software engineering (SBSE) user (researcher or practitioner), do you wonder which multi-objective search algorithm(s) (MOSAs) to use to solve your SE problem? If so, instead of just following the crowd and picking the more commonly ...
On the use of many quality attributes for software refactoring: a many-objective search-based software engineering approach

Search-based software engineering (SBSE) solutions are still not scalable enough to handle high-dimensional objectives space. The majority of existing work treats software engineering problems from a single or bi-objective point of view, where the main ...
Quality assessment of systematic reviews in software engineering: a tertiary study
EASE '15: Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering

Context: The quality of an Systematic Literature Review (SLR) is as good as the quality of the reviewed papers. Hence, it is vital to rigorously assess the papers included in an SLR. There has been no tertiary study aimed at reporting the state of the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE-NIER '18: Proceedings of the 40th International Conference on Software Engineering: New Ideas and Emerging Results

May 2018

130 pages

ISBN:9781450356626

DOI:10.1145/3183399

Conference Chairs:
Andrea Zisman
The Open University
,
Sven Apel
University of Passau, Germany

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 May 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '18

Sponsor:

SIGSOFT
IEEE-CS

ICSE '18: 40th International Conference on Software Engineering

May 27 - June 3, 2018

Gothenburg, Sweden

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

44
Total Citations
View Citations
204
Total Downloads

Downloads (Last 12 months)12
Downloads (Last 6 weeks)1

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Chen TLi M(2023)The Weights Can Be Harmful: Pareto Search versus Weighted Search in Multi-objective Search-based Software EngineeringACM Transactions on Software Engineering and Methodology10.1145/351423332:1(1-40)Online publication date: 13-Feb-2023
https://dl.acm.org/doi/10.1145/3514233
Li MChen T(2023)Methodology and Guidelines for Evaluating Multi-Objective Search-Based Software Engineering2023 IEEE/ACM 45th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)10.1109/ICSE-Companion58688.2023.00096(338-339)Online publication date: May-2023
https://doi.org/10.1109/ICSE-Companion58688.2023.00096
Pallathadka LPallathadka H(2022)Racism and Health: Evidence and Needed ResearchIntegrated Journal for Research in Arts and Humanities10.55544/ijrah.2.6.172:6(128-136)Online publication date: 25-Nov-2022
https://doi.org/10.55544/ijrah.2.6.17
Chen TLi M(2022)Do Performance Aspirations Matter for Guiding Software Configuration Tuning? An Empirical Investigation under Dual Performance ObjectivesACM Transactions on Software Engineering and Methodology10.1145/357185332:3(1-41)Online publication date: 24-Nov-2022
https://dl.acm.org/doi/10.1145/3571853
Li MChen TYao X(2022)How to Evaluate Solutions in Pareto-Based Search-Based Software Engineering: A Critical Review and Methodological GuidanceIEEE Transactions on Software Engineering10.1109/TSE.2020.303610848:5(1771-1799)Online publication date: 1-May-2022
https://dl.acm.org/doi/10.1109/TSE.2020.3036108
Sellami KOuni ASaied MBouktif SMkaouer M(2022)Improving microservices extraction using evolutionary searchInformation and Software Technology10.1016/j.infsof.2022.106996151(106996)Online publication date: Nov-2022
https://doi.org/10.1016/j.infsof.2022.106996
Xiang YYang XHuang HHuang ZLi M(2022)Sampling configurations from software product lines via probability-aware diversification and SAT solvingAutomated Software Engineering10.1007/s10515-022-00348-829:2Online publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1007/s10515-022-00348-8
Imoize AAdedeji OTandiya NShetty S(2021)6G Enabled Smart Infrastructure for Sustainable Society: Opportunities, Challenges, and Research RoadmapSensors10.3390/s2105170921:5(1709)Online publication date: 2-Mar-2021
https://doi.org/10.3390/s21051709
Oni OLetier E(2021)Analyzing Uncertainty in Release Planning: A Method and Experiment for Fixed-Date Release CyclesACM Transactions on Software Engineering and Methodology10.1145/349048731:2(1-39)Online publication date: 24-Dec-2021
https://dl.acm.org/doi/10.1145/3490487
Menéndez HJahangirova GSarro FTonella PClark D(2021)Diversifying Focused Testing for Unit TestingACM Transactions on Software Engineering and Methodology10.1145/344726530:4(1-24)Online publication date: 19-Apr-2021
https://dl.acm.org/doi/10.1145/3447265
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents