research-article

Covariance matrix adaptation evolution strategy based on correlated evolution paths with application to reinforcement learning

Authors:

Oladayo S. Ajani,

Abhishek Kumar,

Rammohan MallipeddiAuthors Info & Claims

Volume 246, Issue C

https://doi.org/10.1016/j.eswa.2024.123289

Published: 15 July 2024 Publication History

Abstract

Proven as an efficient population-based optimization algorithm, Covariance Matrix Adaptation Evolution Strategy (CMA-ES) features two evolution paths, one to update the covariance matrix and the other to adapt its mutation strength. Considering the time and space complexity of CMA-ES, there are several attempts in the literature to realize a single-path algorithm. However, such attempts require altering the original structure of CMA-ES and consequently eliminating some vital features crucial to the overall algorithm performance. In this paper, we show that the two evolution paths of CMA-ES are highly correlated and one can be expressed in terms of the other thus reducing the computational cost of the algorithm while preserving the original algorithmic framework. Based on experimental studies conducted using 30 functions from the IEEE CEC 2014 benchmark suite, the proposed algorithm shows comparable results with the standard CMA-ES as well as five other state-of-the-art CMA-ES variants. Furthermore, it is shown that the proposed algorithm can be applied to policy search in Deep Reinforcement Learning (DRL). Performance results based on selected DRL problems from different application domains prove the efficiency of the proposed algorithm compared to other population-based algorithms often applied for policy search in DRL.

Highlights

•

A correlated evolution path CMAES algorithm is proposed for optimization.

•

The proposed algorithm is evaluated on IEEE CEC 2014 Benchmark.

•

The proposed algorithm is applied in optimal policy search on RL tasks.

•

Comparative performance is demonstrated with other start-of-the-art algorithms.

References

[1]

Ajani O.S., Mallipeddi R., Adaptive evolution strategy with ensemble of mutations for reinforcement learning, Knowledge-Based Systems 245 (2022).

[2]

Arabas J., Jagodziński D., Toward a matrix-free covariance matrix adaptation evolution strategy, IEEE Transactions on Evolutionary Computation 24 (1) (2019) 84–98.

[3]

Beyer H.-G., Schwefel H.-P., Evolution strategies – A comprehensive introduction, Natural Computing 1 (2004) 3–52.

[4]

Beyer H.-G., Sendhoff B., Covariance matrix adaptation revisited - The CMSA evolution strategy-, in: PPSN, 2008.

[5]

Beyer H.-G., Sendhoff B., Simplify your covariance matrix adaptation evolution strategy, IEEE Transactions on Evolutionary Computation 21 (5) (2017) 746–759.

[6]

Chrabaszcz P., Loshchilov I., Hutter F., Back to basics: Benchmarking canonical evolution strategies for playing atari, 2018, arXiv preprint abs/1802.08842, (2018).

[7]

Conti E., Madhavan V., Such F.P., Lehman J., Stanley K.O., Clune J., Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-seeking agents, in: NeurIPS, 2018.

[8]

Coumans E., Bai Y., PyBullet, a python module for physics simulation for games, robotics and machine learning, 2016, http://pybullet.org.

[9]

Garcia S., Herrera F., An extension on ”Statistical Comparisons of Classifiers over Multiple Data Sets” for all pairwise comparisons, Journal of Machine Learning Research 9 (2008) 2677–2694.

[10]

Ha D., Evolving stable strategies, blog.otoro.net (2017) URL http://blog.otoro.net/2017/11/12/evolving-stable-strategies/.

[11]

Hansen N., The CMA evolution strategy: A comparing review, in: Towards a new evolutionary computation: advances in the estimation of distribution algorithms, 2006, pp. 75–102.

[12]

Hansen N., The CMA evolution strategy: A tutorial, 2016, arXiv abs/1604.00772.

[13]

Hansen N., Auger A., Principled design of continuous stochastic search: From theory to practice, in: Theory and principled methods for the design of metaheuristics, 2014, pp. 145–180.

[14]

Hansen N., Müller S.D., Koumoutsakos P., Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES), Evolutionary Computation 11 (1) (2003) 1–18.

Digital Library

[15]

Hansen N., Ostermeier A., Completely derandomized self-adaptation in evolution strategies, Evolutionary Computation 9 (2) (2001) 159–195.

Digital Library

[16]

Holm S., A simple sequentially rejective multiple test procedure, Scandinavian Journal of Statistics 6 (1979) 65–70.

[17]

Igel C., Suttorp T., Hansen N., A computational efficient covariance matrix update and a (1+1)-CMA for evolution strategies, in: Proceedings of the 8th annual conference on genetic and evolutionary computation, Association for Computing Machinery, New York, NY, USA, 2006, pp. 453–460,.

Digital Library

[18]

Jastrebski, G., & Arnold, D. (2006). Improving Evolution Strategies through Active Covariance Matrix Adaptation. In 2006 IEEE international conference on evolutionary computation (pp. 2814–2821).

[19]

Kumar A., Das S., Mallipeddi R., A reference vector-based simplified covariance matrix adaptation evolution strategy for constrained global optimization, IEEE Transactions on Cybernetics (2020) 1–14.

[20]

Li Z., Zhang Q., An efficient rank-1 update for cholesky CMA-ES using auxiliary evolution path, in: 2017 IEEE congress on evolutionary computation, IEEE, 2017, pp. 913–920.

[21]

Li Z., Zhang Q., A simple yet efficient evolution strategy for large-scale black-box optimization, IEEE Transactions on Evolutionary Computation 22 (5) (2017) 637–646.

Digital Library

[22]

Li Z., Zhang Q., Variable metric evolution strategies by mutation matrix adaptation, Information Sciences 541 (2020) 136–151.

[23]

Liang J.-C., Qu B., Suganthan P.N., Problem definitions and evaluation criteria for the CEC 2014 special session and competition on single objective real-parameter numerical optimization, 2014.

[24]

Loshchilov I., LM-CMA: An alternative to L-BFGS for large-scale black box optimization, Evolutionary Computation 25 (1) (2017) 143–171.

[25]

Loshchilov I., Glasmachers T., Beyer H.-G., Large scale black-box optimization by limited-memory matrix adaptation, IEEE Transactions on Evolutionary Computation 23 (2) (2018) 353–358.

[26]

Müller, C. L., & Sbalzarini, I. F. (2010). Gaussian Adaptation as a unifying framework for continuous black-box optimization and adaptive Monte Carlo sampling. In IEEE congress on evolutionary computation (pp. 1–8).

[27]

Qu, X., Ong, Y.-S., Hou, Y., & Shen, X. (2019). Memetic Evolution Strategy for Reinforcement Learning. In 2019 IEEE congress on evolutionary computation (pp. 1922–1928).

[28]

Salimans T., Ho J., Chen X., Sutskever I., Evolution strategies as a scalable alternative to reinforcement learning, 2017, ArXiv.

[29]

Schwefel H.-P.P., Evolution and optimum seeking: the sixth generation, John Wiley & Sons, Inc., USA, 1993.

[30]

Sehnke F., Osendorfer C., Rückstieß T., Graves A., Peters J., Schmidhuber J., Parameter-exploring policy gradients, Neural Networks 23 (4) (2010) 551–559.

[31]

Such F.P., Madhavan V., Conti E., Lehman J., Stanley K.O., Clune J., Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning, 2017, arXiv preprint, abs/1712.06567, (2017).

[32]

Wierstra D., Schaul T., Glasmachers T., Sun Y., Peters J., Schmidhuber J., Natural evolution strategies, Journal of Machine Learning Research 15 (1) (2014) 949–980.

[33]

Williams R.J., Simple statistical gradient-following algorithms for connectionist reinforcement learning, Machine Learning 8 (1992) 229–256.

Digital Library

Cited By

Aboyeji EAjani OMallipeddi R(2024)Covariance matrix adaptation evolution strategy based on ensemble of mutations for parking navigation and maneuver of autonomous vehiclesExpert Systems with Applications: An International Journal10.1016/j.eswa.2024.123565249:PAOnline publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1016/j.eswa.2024.123565

Recommendations

Self-adaptive surrogate-assisted covariance matrix adaptation evolution strategy
GECCO '12: Proceedings of the 14th annual conference on Genetic and evolutionary computation

This paper presents a novel mechanism to adapt surrogate-assisted population-based algorithms. This mechanism is applied to ACM-ES, a recently proposed surrogate-assisted variant of CMA-ES. The resulting algorithm, ^s*ACM-ES, adjusts online the ...
Adaptive evolution strategy with ensemble of mutations for Reinforcement Learning
Abstract
Evolving the weights of learning networks through evolutionary computation (neuroevolution) has proven scalable over a range of challenging Reinforcement Learning (RL) control tasks. However, similar to most black-box optimization ...
Asynchronous differential evolution with adaptive correlation matrix
GECCO '13: Proceedings of the 15th annual conference on Genetic and evolutionary computation

Differential evolution (DE) is an efficient algorithm to solve global optimization problems. It has a simple internal structure and uses a few control parameters. In this paper we incorporate crossover based on adaptive correlation matrix into ...

Comments

Information & Contributors

Information

Published In

cover image Expert Systems with Applications: An International Journal

Expert Systems with Applications: An International Journal Volume 246, Issue C

Jul 2024

1587 pages

Issue’s Table of Contents

Elsevier Ltd.

Publisher

Pergamon Press, Inc.

United States

Publication History

Published: 15 July 2024

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 11 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Aboyeji EAjani OMallipeddi R(2024)Covariance matrix adaptation evolution strategy based on ensemble of mutations for parking navigation and maneuver of autonomous vehiclesExpert Systems with Applications: An International Journal10.1016/j.eswa.2024.123565249:PAOnline publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1016/j.eswa.2024.123565

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents