Abstract
Algorithm selection wizards are effective and versatile tools that automatically select an optimization algorithm given high-level information about the problem and available computational resources, such as number and type of decision variables, maximal number of evaluations, possibility to parallelize evaluations, etc. State-of-the-art algorithm selection wizards are complex and difficult to improve. We propose in this work the use of automated configuration methods for improving their performance by finding better configurations of the algorithms that compose them. In particular, we use elitist iterated racing (irace) to find CMA configurations for specific artificial benchmarks that replace the hand-crafted CMA configurations currently used in the NGOpt wizard provided by the Nevergrad platform. We discuss in detail the setup of irace for the purpose of generating configurations that work well over the diverse set of problem instances within each benchmark. Our approach improves the performance of the NGOpt wizard, even on benchmark suites that were not part of the tuning by irace.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Aziz-Alaoui, A., Doerr, C., Dréo, J.: Towards large scale automated algorithm design by integrating modular benchmarking frameworks. In: Chicano, F., Krawiec, K. (eds.) Proceedings of the Genetic and Evolutionary Computation Conference Companion, GECCO Companion 2021, New York, NY, pp. 1365–1374. ACM Press (2021). https://doi.org/10.1145/3449726.3463155
Bezerra, L.C.T., López-Ibáñez, M., Stützle, T.: Automatically designing state-of-the-art multi- and many-objective evolutionary algorithms. Evol. Comput. 28(2), 195–226 (2020). https://doi.org/10.1162/evco_a_00263
Birattari, M.: Tuning Metaheuristics: A Machine Learning Perspective, Studies in Computational Intelligence, vol. 197. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-00483-4
Cartis, C., Fiala, J., Marteau, B., Roberts, L.: Improving the flexibility and robustness of model-based derivative-free optimization solvers (2018)
Hansen, N., Auger, A., Finck, S., Ros, R.: Real-parameter black-box optimization benchmarking 2009: Experimental setup. Technical report, RR-6828, INRIA, France (2009). https://hal.inria.fr/inria-00362633/document
Hansen, N., Auger, A., Ros, R., Mersmann, O., Tušar, T., Brockhoff, D.: COCO: a platform for comparing continuous optimizers in a black-box setting. Optim. Meth. Software 36(1), 1–31 (2020). https://doi.org/10.1080/10556788.2020.1808977
Hansen, N., Ostermeier, A.: Completely derandomized self-adaptation in evolution strategies. Evol. Comput. 9(2), 159–195 (2001). https://doi.org/10.1162/106365601750190398
Hoos, H.H.: Automated algorithm configuration and parameter tuning. In: Hamadi, Y., Monfroy, E., Saubion, F. (eds.) Autonomous Search, pp. 37–71. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-21434-9_3
Huang, C., Li, Y., Yao, X.: A survey of automatic parameter tuning methods for metaheuristics. IEEE Trans. Evol. Comput. 24(2), 201–216 (2020). https://doi.org/10.1109/TEVC.2019.2921598
Johnson, S.G.: The nlopt nonlinear-optimization package (1994). http://github.com/stevengj/nlopt
Kerschke, P., Hoos, H.H., Neumann, F., Trautmann, H.: Automated algorithm selection: survey and perspectives. Evol. Comput. 27(1), 3–45 (2019). https://doi.org/10.1162/evco_a_00242
KhudaBukhsh, A.R., Xu, L., Hoos, H.H., Leyton-Brown, K.: SATenstein: automatically building local search SAT solvers from components. Artif. Intell. 232, 20–42 (2016). https://doi.org/10.1016/j.artint.2015.11.002
Liao, T., Molina, D., Stützle, T.: Performance evaluation of automatically tuned continuous optimizers on different benchmark sets. Appl. Soft Comput. 27, 490–503 (2015)
Liao, T., Montes de Oca, M.A., Stützle, T.: Computational results for an automatically tuned CMA-ES with increasing population size on the CEC 2005 benchmark set. Soft Comput. 17(6), 1031–1046 (2013). https://doi.org/10.1007/s00500-012-0946-x
Liao, T., Stützle, T., Montes de Oca, M.A., Dorigo, M.: A unified ant colony optimization algorithm for continuous optimization. Eur. J. Oper. Res. 234(3), 597–609 (2014)
López-Ibáñez, M., Dubois-Lacoste, J., Pérez Cáceres, L., Stützle, T., Birattari, M.: The irace pacskage: iterated racing for automatic algorithm configuration. Oper. Res. Perspect. 3, 43–58 (2016). https://doi.org/10.1016/j.orp.2016.09.002
López-Ibáñez, M., Liao, T., Stützle, T.: On the anytime behavior of IPOP-CMA-ES. In: Coello, C.A.C., Cutello, V., Deb, K., Forrest, S., Nicosia, G., Pavone, M. (eds.) PPSN 2012. LNCS, vol. 7491, pp. 357–366. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-32937-1_36
López-Ibáñez, M., Stützle, T.: The automatic design of multi-objective ant colony optimization algorithms. IEEE Trans. Evol. Comput. 16(6), 861–875 (2012). https://doi.org/10.1109/TEVC.2011.2182651
Mascia, F., López-Ibáñez, M., Dubois-Lacoste, J., Stützle, T.: Grammar-based generation of stochastic local search heuristics through automatic algorithm configuration tools. Comput. Oper. Res. 51, 190–199 (2014). https://doi.org/10.1016/j.cor.2014.05.020
Meunier, L., et al.: Black-box optimization revisited: improving algorithm selection wizards through massive benchmarking. IEEE Trans. Evol. Comput. 26(3), 490–500 (2022). https://doi.org/10.1109/TEVC.2021.3108185
Pagnozzi, F., Stützle, T.: Automatic design of hybrid stochastic local search algorithms for permutation flowshop problems. Eur. J. Oper. Res. 276, 409–421 (2019). https://doi.org/10.1016/j.ejor.2019.01.018
Rapin, J., Teytaud, O.: Nevergrad: a gradient-free optimization platform (2018). https://github.com/FacebookResearch/Nevergrad
Rice, J.R.: The algorithm selection problem. Adv. Comput. 15, 65–118 (1976)
Ros, R., Hansen, N.: A simple modification in CMA-ES achieving linear time and space complexity. In: Rudolph, G., Jansen, T., Beume, N., Lucas, S., Poloni, C. (eds.) PPSN 2008. LNCS, vol. 5199, pp. 296–305. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-87700-4_30
Schede, E., et al.: A survey of methods for automated algorithm configuration (2022). https://doi.org/10.48550/ARXIV.2202.01651
Styles, J., Hoos, H.H.: Ordered racing protocols for automatically configuring algorithms for scaling performance. In: Blum, C., Alba, E. (eds.) Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2013, New York, NY, pp. 551–558. ACM Press (2013). ISBN 978-1-4503-1963-8, https://doi.org/10.1145/2463372.2463438
Videau, M., Leite, A., Teytaud, O., Schoenauer, M.: Multi-objective genetic programming for explainable reinforcement learning. In: Medvet, E., Pappa, G., Xue, B. (eds.) EuroGP 2022. LNCS, vol. 13223, pp. 256–281. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-02056-8_18
Xu, L., Hoos, H.H., Leyton-Brown, K.: Hydra: automatically configuring algorithms for portfolio-based selection. In: Fox, M., Poole, D. (eds.) Proceedings of the AAAI Conference on Artificial Intelligence. AAAI Press (2010)
Xu, L., Hutter, F., Hoos, H.H., Leyton-Brown, K.: SATzilla: portfolio-based algorithm selection for SAT. J. Artif. Intell. Res. 32, 565–606 (2008). https://doi.org/10.1613/jair.2490
Acknowledgments
M. López-Ibáñez is a “Beatriz Galindo” Senior Distinguished Researcher (BEAGAL 18/00053) funded by the Spanish Ministry of Science and Innovation (MICINN). C. Doerr is supported by the Paris Ile-de-France Region (AlgoSelect) and by the INS2I institute of CNRS (RandSearch). T. Eftimov, A. Nikolikj, and G. Cenikj is supported by the Slovenian Research Agency: research core fundings No. P2-0098 and project No. N2-0239. G. Cenikj is also supported by the Ad Futura grant for postgraduate study.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Trajanov, R. et al. (2022). Improving Nevergrad’s Algorithm Selection Wizard NGOpt Through Automated Algorithm Configuration. In: Rudolph, G., Kononova, A.V., Aguirre, H., Kerschke, P., Ochoa, G., Tušar, T. (eds) Parallel Problem Solving from Nature – PPSN XVII. PPSN 2022. Lecture Notes in Computer Science, vol 13398. Springer, Cham. https://doi.org/10.1007/978-3-031-14714-2_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-14714-2_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-14713-5
Online ISBN: 978-3-031-14714-2
eBook Packages: Computer ScienceComputer Science (R0)