research-article

SELECTOR: selecting a representative benchmark suite for reproducible statistical comparison

Authors:

Gjorgjina Cenikj,

Ryan Dieter Lang,

Andries Petrus Engelbrecht,

Peter Korošec,

Tome EftimovAuthors Info & Claims

GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference

Pages 620 - 629

https://doi.org/10.1145/3512290.3528809

Published: 08 July 2022 Publication History

Abstract

Fair algorithm evaluation is conditioned on the existence of high-quality benchmark datasets that are non-redundant and are representative of typical optimization scenarios. In this paper, we evaluate three heuristics for selecting diverse problem instances which should be involved in the comparison of optimization algorithms in order to ensure robust statistical algorithm performance analysis. The first approach employs clustering to identify similar groups of problem instances and subsequent sampling from each cluster to construct new benchmarks, while the other two approaches use graph algorithms for identifying dominating and maximal independent sets of nodes. We demonstrate the applicability of the proposed heuristics by performing a statistical performance analysis of five portfolios consisting of three optimization algorithms on five of the most commonly used optimization benchmarks.

The results indicate that the statistical analyses of the algorithms' performance, conducted on each benchmark separately, produce conflicting outcomes, which can be used to give a false indication of the superiority of one algorithm over another. On the other hand, when the analysis is conducted on the problem instances selected with the proposed heuristics, which uniformly cover the problem landscape, the statistical outcomes are robust and consistent.

References

[1]

Robert B. Allan and Renu Laskar. 1978. On domination and independent domination numbers of a graph. Discrete Mathematics 23, 2 (1978), 73--76.

[2]

B. Beachkofski and R. Grandhi. 2002. Improved Distributed Hypercube Sampling. In 43rd AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference. American Institute of Aeronautics and Astronautics.

[3]

Borja Calvo and Guzman Santafe. 2015. scmamp: Statistical Comparison of Multiple Algorithms in Multiple Problems. The R Journal Accepted for publication (2015).

[4]

Gavin C. Cawley and Nicola L. C. Talbot. 2010. On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation. Journal of Machine Learning Research 11, 70 (2010), 2079--2107. http://jmlr.org/papers/v11/cawley10a.html

Digital Library

[5]

Lee A Christie, Alexander EI Brownlee, and John R Woodward. 2018. Investigating benchmark correlations when comparing algorithms with parameter tuning. In Proc. of the Genetic and Evolutionary Computation Conference Companion. 209--210.

Digital Library

[6]

Janez Demšar. 2006. Statistical Comparisons of Classifiers over Multiple Data Sets. 7 (dec 2006), 1--30.

[7]

Tome Eftimov, Peter Korošec, and Barbara Koroušić Seljak. 2017. A novel approach to statistical comparison of meta-heuristic stochastic optimization algorithms using deep statistics. Information Sciences 417 (2017), 186--215.

Digital Library

[8]

Tome Eftimov, Gorjan Popovski, Quentin Renau, Peter Korošec, and Carola Doerr. 2020. Linear matrix factorization embeddings for single-objective optimization landscapes. In 2020 IEEE Symposium Series on Computational Intelligence (SSCI). IEEE, 775--782.

[9]

Salvador García, Daniel Molina, Manuel Lozano, and Francisco Herrera. 2009. A study on the use of non-parametric tests for analyzing the evolutionary algorithms' behaviour: a case study on the CEC'2005 special session on real parameter optimization. Journal of Heuristics 15, 6 (2009), 617--644.

Digital Library

[10]

Aric A. Hagberg, Daniel A. Schult, and Pieter J. Swart. 2008. Exploring Network Structure, Dynamics, and Function using NetworkX. (2008), 11 -- 15.

[11]

Nikolaus Hansen, Anne Auger, Raymond Ros, Olaf Mersmann, Tea Tušar, and Dimo Brockhoff. 2020. COCO: A platform for comparing continuous optimizers in a black-box setting. Optimization Methods and Software (2020), 1--31.

[12]

Nikolaus Hansen, Steffen Finck, Raymond Ros, and Anne Auger. 2009. Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions. Research Report RR-6829. INRIA. https://hal.inria.fr/inria-00362633

[13]

Nikolaus Hansen and Andreas Ostermeier. 2001. Completely Derandomized Self-Adaptation in Evolution Strategies. Evolutionary Computation 9, 2 (2001), 159--195.

Digital Library

[14]

Peter Jeavons, Alex D. Scott, and Lei Xu. 2016. Feedback from nature: simple randomised distributed algorithms for maximal independent set selection and greedy colouring. Distributed Computing 29 (2016), 377--393.

Digital Library

[15]

J. Kennedy and R. Eberhart. 1995. Particle swarm optimization. In Proc. of ICNN'95 - International Conference on Neural Networks, Vol. 4. 1942--1948 vol.4.

[16]

P. Kerschke, M. Preuss, S. Wessing, and H. Trautmann. 2015. Detecting Funnel Structures by Means of Exploratory Landscape Analysis. In Proc. of Genetic and Evolutionary Computation Conference (GECCO'15). ACM, 265--272.

Digital Library

[17]

Pascal Kerschke and Heike Trautmann. 2019. Comprehensive Feature-Based Landscape Analysis of Continuous and Constrained Optimization Problems Using the R-package flacco. In Applications in Statistical Computing - From Music Data Analysis to Industrial Quality Improvement, Nadja Bauer, Katja Ickstadt, Karsten Lübke, Gero Szepannek, Heike Trautmann, and Maurizio Vichi (Eds.). Springer, 93 -- 123.

[18]

Benjamin Lacroix and John McCall. 2019. Limitations of Benchmark Sets and Landscape Features for Algorithm Selection and Performance Prediction. In Proc. of Genetic and Evolutionary Computation (GECCO'19, Companion). ACM, 261--262.

Digital Library

[19]

Ryan Dieter Lang and Andries Petrus Engelbrecht. 2021. An Exploratory Landscape Analysis-Based Benchmark Suite. Algorithms 14, 3 (2021).

[20]

J.J. Liang, B. Qu, and Ponnuthurai Suganthan. 2013. Problem definitions and evaluation criteria for the CEC 2014 special session and competition on single objective real-parameter numerical optimization. Computational Intelligence Laboratory, Zhengzhou University, Zhengzhou China and Technical Report, Nanyang Technological University, Singapore. (12 2013).

[21]

J.J. Liang, B.Y. Qu, P.N. Suganthan, and Q. Chen. 2014. Problem definitions and evaluation criteria for the CEC 2015 competition on learning-based real-parameter single objective optimization. Computational Intelligence Laboratory, Zhengzhou University, Zhengzhou China and Technical Report, Nanyang Technological University, Singapore. (2014).

[22]

J.J. Liang, B. Qu, Ponnuthurai Suganthan, and Alfredo Hernández-Díaz. 2013. Problem Definitions and Evaluation Criteria for the CEC 2013 Special Session on Real-Parameter Optimization. Computational Intelligence Laboratory, Zhengzhou University, Zhengzhou China and Technical Report, Nanyang Technological University, Singapore. (01 2013).

[23]

M. Lunacek and D. Whitley. 2006. The dispersion metric and the CMA evolution strategy. In Proc. of Genetic and Evolutionary Computation Conference (GECCO'06). ACM, 477.

Digital Library

[24]

Olaf Mersmann, Bernd Bischl, Heike Trautmann, Mike Preuss, Claus Weihs, and Günter Rudolph. 2011. Exploratory landscape analysis. In Proc. of Genetic and Evolutionary Computation Conference (GECCO'11). ACM, 829--836.

Digital Library

[25]

Laurent Meunier, Herilalaina Rakotoarison, Pak Kan Wong, Baptiste Roziere, Jeremy Rapin, Olivier Teytaud, Antoine Moreau, and Carola Doerr. 2021. Blackbox optimization revisited: Improving algorithm selection wizards through massive benchmarking. IEEE Transactions on Evolutionary Computation (2021).

[26]

M.A. Muñoz, M. Kirley, and S.K. Halgamuge. 2015. Exploratory Landscape Analysis of Continuous Space Optimization Problems Using Information Content. IEEE Transactions on Evolutionary Computation 19, 1 (2015), 74--87.

Digital Library

[27]

Mario A. Muñoz and Kate Smith-Miles. 2020. Generating New Space-Filling Test Instances for Continuous Black-Box Optimization. Evolutionary Computation 28, 3 (09 2020), 379--404. arXiv:https://direct.mit.edu/evco/article-pdf/28/3/379/1858988/evco_a_00262.pdf

Digital Library

[28]

P. Nemenyi. 1963. Distribution-free Multiple Comparisons. Princeton University. https://books.google.si/books?id=nhDMtgAACAAJ

[29]

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research 12 (2011), 2825--2830.

Digital Library

[30]

Thorsten Pohlert. 2014. The Pairwise Multiple Comparison of Mean Ranks Package (PMCMR). https://CRAN.R-project.org/package=PMCMR R package.

[31]

J. Rapin and O. Teytaud. 2018. Nevergrad - A gradient-free optimization platform. https://GitHub.com/FacebookResearch/Nevergrad.

[32]

Ali Seyed Shirkhorshidi, Saeed Aghabozorgi, and Teh Ying Wah. 2015. A comparison study on similarity and dissimilarity measures in clustering continuous data. PloS one 10, 12 (2015), e0144059.

[33]

Urban Škvorc, Tome Eftimov, and Peter Korošec. 2020. Understanding the problem space in single-objective numerical optimization using exploratory landscape analysis. Applied Soft Computing 90 (May 2020), 106138.

[34]

Urban Škvorc, Tome Eftimov, and Peter Korošec. 2020. Understanding the problem space in single-objective numerical optimization using exploratory landscape analysis. Applied Soft Computing 90 (2020), 106138.

[35]

Urban Skvorc, Tome Eftimov, and Peter Korosec. 2021. A Complementarity Analysis of the COCO Benchmark Problems and Artificially Generated Problems. CoRR abs/2104.13060 (2021). arXiv:2104.13060 https://arxiv.org/abs/2104.13060

[36]

Kate Smith-Miles, Davaatseren Baatar, Brendan Wreford, and Rhyd Lewis. 2014. Towards objective measures of algorithm performance across instance space. Computers & Operations Research 45 (2014), 12--24.

Digital Library

[37]

Rainer Storn and Kenneth Price. 1997. Differential Evolution - A Simple and Efficient Heuristic for global Optimization over Continuous Spaces. Journal of Global Optimization 11, 4 (1997), 341--359.

Digital Library

[38]

Thomas Weise, Yan Chen, Xinlu Li, and Zhize Wu. 2020. Selecting a diverse set of benchmark instances from a tunable model problem for black-box discrete optimization algorithms. Appl. Soft Comput. 92 (2020), 106269.

[39]

Guohua Wu, Rammohan Mallipeddi, and Ponnuthurai Suganthan. 2016. Problem Definitions and Evaluation Criteria for the CEC 2017 Competition and Special Session on Constrained Single Objective Real-Parameter Optimization. Computational Intelligence Laboratory, Zhengzhou University, Zhengzhou China and Technical Report, Nanyang Technological University, Singapore. (10 2016).

[40]

Estefania Yap, Mario A. Munoz, Kate Smith-Miles, and Arnaud Liefooghe. 2020. Instance Space Analysis of Combinatorial Multi-objective Optimization Problems. In 2020 IEEE Congress on Evolutionary Computation (CEC). IEEE.

Digital Library

[41]

Bo Yuan and Marcus Gallagher. 2004. Statistical Racing Techniques for Improved Empirical Evaluation of Evolutionary Algorithms. In Parallel Problem Solving from Nature - PPSN VIII, Xin Yao, Edmund K. Burke, José A. Lozano, Jim Smith, Juan Julián Merelo-Guervós, John A. Bullinaria, Jonathan E. Rowe, Peter Tiňo, Ata Kabán, and Hans-Paul Schwefel (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 172--181.

[42]

Yong-Wei Zhang and Saman K Halgamuge. 2019. Similarity of Continuous Optimization Problems from the Algorithm Performance Perspective. In 2019 IEEE Congress on Evolutionary Computation (CEC). IEEE, 2949--2957.

Digital Library

[43]

Urban Škvorc, Tome Eftimov, and Peter Korošec. 2022. Transfer Learning Analysis of Multi-Class Classification for Landscape-Aware Algorithm Selection. Mathematics 10, 3 (2022).

Cited By

Vermetten DYe FBäck TDoerr C(2024)MA-BBOB: A Problem Generator for Black-Box Optimization Using Affine Combinations and ShiftsACM Transactions on Evolutionary Learning and Optimization10.1145/3673908Online publication date: 21-Jun-2024
https://dl.acm.org/doi/10.1145/3673908
Gallagher MMunoz MLi XHandl J(2024)Towards an Improved Understanding of Features for More Interpretable Landscape AnalysisProceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3638530.3654301(135-138)Online publication date: 14-Jul-2024
https://dl.acm.org/doi/10.1145/3638530.3654301
Cenikj GPetelin GEftimov TLi XHandl J(2024)TransOptAS: Transformer-Based Algorithm Selection for Single-Objective OptimizationProceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3638530.3654191(403-406)Online publication date: 14-Jul-2024
https://dl.acm.org/doi/10.1145/3638530.3654191
Show More Cited By

Index Terms

SELECTOR: selecting a representative benchmark suite for reproducible statistical comparison
1. Theory of computation
  1. Design and analysis of algorithms
    1. Mathematical optimization
      1. Continuous optimization

Recommendations

GECCO'16 Black-Box Optimization Benchmarking Workshop (BBOB-2016): Workshop Chairs' Welcome Message
GECCO '16 Companion: Proceedings of the 2016 on Genetic and Evolutionary Computation Conference Companion
Comparison Of Three Versions Of Whale Optimization Algorithm (WOA) On The Bbob Test Suite
GECCO '23 Companion: Proceedings of the Companion Conference on Genetic and Evolutionary Computation

This paper benchmarks the canonical version of Whale Optimization Algorithm (WOA) with two recent adaptations: "Enhanced WOA (E-WOA)" and "modified Symbiotic organism search - Differential Evolution - WOA (m-SDWOA)" with the desire to research the ...
Black-box optimization benchmarking for noiseless function testbed using particle swarm optimization
GECCO '09: Proceedings of the 11th Annual Conference Companion on Genetic and Evolutionary Computation Conference: Late Breaking Papers

This paper benchmarks the Particle Swarm Optimization (PSO) algorithm using the noise-free BBOB 2009 testbed.

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference

July 2022

1472 pages

ISBN:9781450392372

DOI:10.1145/3512290

Editor:
Jonathan E. Fieldsend
University of Exeter
,
General Chair:
Markus Wagner
The University of Adelaide

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGEVO: ACM Special Interest Group on Genetic and Evolutionary Computation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 July 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Javna Agencija za Raziskovalno Dejavnost RS
Ad Futura grant for postgraduate study

Conference

GECCO '22

Sponsor:

SIGEVO

GECCO '22: Genetic and Evolutionary Computation Conference

July 9 - 13, 2022

Massachusetts, Boston

Acceptance Rates

Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
96
Total Downloads

Downloads (Last 12 months)33
Downloads (Last 6 weeks)7

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Vermetten DYe FBäck TDoerr C(2024)MA-BBOB: A Problem Generator for Black-Box Optimization Using Affine Combinations and ShiftsACM Transactions on Evolutionary Learning and Optimization10.1145/3673908Online publication date: 21-Jun-2024
https://dl.acm.org/doi/10.1145/3673908
Gallagher MMunoz MLi XHandl J(2024)Towards an Improved Understanding of Features for More Interpretable Landscape AnalysisProceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3638530.3654301(135-138)Online publication date: 14-Jul-2024
https://dl.acm.org/doi/10.1145/3638530.3654301
Cenikj GPetelin GEftimov TLi XHandl J(2024)TransOptAS: Transformer-Based Algorithm Selection for Single-Objective OptimizationProceedings of the Genetic and Evolutionary Computation Conference Companion10.1145/3638530.3654191(403-406)Online publication date: 14-Jul-2024
https://dl.acm.org/doi/10.1145/3638530.3654191
Dietrich KVermetten DDoerr CKerschke PLi XHandl J(2024)Impact of Training Instance Selection on Automated Algorithm Selection Models for Numerical Black-box OptimizationProceedings of the Genetic and Evolutionary Computation Conference10.1145/3638529.3654100(1007-1016)Online publication date: 14-Jul-2024
https://dl.acm.org/doi/10.1145/3638529.3654100
Petelin GCenikj G(2024)Random Filter Mappings as Optimization Problem Feature ExtractorsIEEE Access10.1109/ACCESS.2024.346872312(143554-143571)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3468723
Eschwege DEngelbrecht A(2024)Belief space-guided approach to self-adaptive particle swarm optimizationSwarm Intelligence10.1007/s11721-023-00232-518:1(31-78)Online publication date: 31-Jan-2024
https://doi.org/10.1007/s11721-023-00232-5
Nikolikj ACenikj GIspirova GVermetten DLang REngelbrecht ADoerr CKorošec PEftimov TSilva SPaquete L(2023)Assessing the Generalizability of a Performance Predictive ModelProceedings of the Companion Conference on Genetic and Evolutionary Computation10.1145/3583133.3590617(311-314)Online publication date: 15-Jul-2023
https://dl.acm.org/doi/10.1145/3583133.3590617
Cenikj GPetelin GDoerr CKorošec PEftimov TSilva SPaquete L(2023)DynamoRep: Trajectory-Based Population Dynamics for Classification of Black-box Optimization ProblemsProceedings of the Genetic and Evolutionary Computation Conference10.1145/3583131.3590401(813-821)Online publication date: 15-Jul-2023
https://dl.acm.org/doi/10.1145/3583131.3590401
Petelin GCenikj G(2023)How Far Out of Distribution Can We Go With ELA Features and Still Be Able to Rank Algorithms?2023 IEEE Symposium Series on Computational Intelligence (SSCI)10.1109/SSCI52147.2023.10371880(341-346)Online publication date: 5-Dec-2023
https://doi.org/10.1109/SSCI52147.2023.10371880
Chandrasekaran SBartz-Beielstein T(2023)A Robust Statistical Framework for the Analysis of the Performances of Stochastic Optimization Algorithms Using the Principles of SeverityApplications of Evolutionary Computation10.1007/978-3-031-30229-9_28(426-441)Online publication date: 12-Apr-2023
https://dl.acm.org/doi/10.1007/978-3-031-30229-9_28
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents