Parametric-Task MAP-Elites
Pages 68 - 77
Abstract
Optimizing a set of functions simultaneously by leveraging their similarity is called multi-task optimization. Current black-box multi-task algorithms only solve a finite set of tasks, even when the tasks originate from a continuous space. In this paper, we introduce Parametric-Task MAP-Elites (PT-ME), a new black-box algorithm for continuous multi-task optimization problems. This algorithm (1) solves a new task at each iteration, effectively covering the continuous space, and (2) exploits a new variation operator based on local linear regression. The resulting dataset of solutions makes it possible to create a function that maps any task parameter to its optimal solution. We show that PT-ME outperforms all baselines, including the deep reinforcement learning algorithm PPO on two parametric-task toy problems and a robotic problem in simulation.
References
[1]
Ram Agrawal, Kalyanmoy Deb, and Ram Agrawal. 2000. Simulated Binary Crossover for Continuous Search Space. Complex Systems 9 (06 2000).
[2]
Timothée Anne and Jean-Baptiste Mouret. 2023. Multi-Task Multi-Behavior MAP-Elites. In Proceedings of the Companion Conference on Genetic and Evolutionary Computation (Lisbon, Portugal) (GECCO '23 Companion). Association for Computing Machinery, New York, NY, USA, 111--114.
[3]
Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. 2017. Deep Reinforcement Learning: A Brief Survey. IEEE Signal Processing Magazine 34, 6 (2017), 26--38.
[4]
Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. 2002. Finite-time Analysis of the Multiarmed Bandit Problem. Machine Learning 47 (05 2002), 235--256.
[5]
Kavitesh Kumar Bali, Abhishek Gupta, Yew-Soon Ong, and Puay Siew Tan. 2021. Cognizant Multitasking in Multiobjective Multifactorial Evolution: MO-MFEA-II. IEEE Transactions on Cybernetics 51, 4 (2021), 1784--1796.
[6]
Kavitesh Kumar Bali, Yew-Soon Ong, Abhishek Gupta, and Puay Siew Tan. 2020. Multifactorial Evolutionary Algorithm With Online Transfer Parameter Estimation: MFEA-II. IEEE Transactions on Evolutionary Computation 24, 1 (2020), 69--83.
[7]
Adrien Baranes and Pierre-Yves Oudeyer. 2013. Active learning of inverse models with intrinsically motivated goal exploration in robots. Robotics and Autonomous Systems 61, 1 (2013), 49--73.
[8]
S. Barnett. 1968. A simple class of parametric linear programming problems. Operations Research 16, 6 (1968), 1160--1165.
[9]
Eric Brochu, Vlad M. Cora, and Nando de Freitas. 2010. A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning. CoRR abs/1012.2599 (2010). arXiv:1012.2599 http://arxiv.org/abs/1012.2599
[10]
Wenxue Chen, Changsheng Gao, and Wuxing Jing. 2023. Proximal policy optimization guidance algorithm for intercepting near-space maneuvering targets. Aerospace Science and Technology 132 (2023), 108031.
[11]
Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean-Baptiste Mouret. 2015. Robots that can adapt like animals. Nature 521, 7553 (2015), 503--507.
[12]
Weijing Dai, Zhenkun Wang, and Ke Xue. 2022. System-in-package design using multi-task memetic learning and optimization. Memetic Computing 14 (03 2022), 1--15.
[13]
Eloïse Dalin, Ivan Bergonzani, Timothée Anne, Serena Ivaldi, and Jean-Baptiste Mouret. 2021. Whole-body teleoperation of the Talos humanoid robot: preliminary results. In ICRA 2021 - 5th Workshop on Teleoperation of Dynamic Legged Robots in Real Scenarios. Xi'an / Virtual, China, https://hal.inria.fr/hal-03245005
[14]
Boris Delaunay. 1928. Sur la sphere vide. In Proceedings of the Mathematics, Toronto. Toronto, 695--700. 11-16 August 1924.
[15]
Qiang Du, Vance Faber, and Max Gunzburger. 1999. Centroidal Voronoi Tessellations: Applications and Algorithms. SIAM Rev. 41, 4 (1999), 637--676. arXiv:https://doi.org/10.1137/S0036144599352836
[16]
Pinky Dua, K Kouramas, Vivek Dua, and Efstratios N Pistikopoulos. 2008. MPC on a chip---Recent advances on the application of multi-parametric model-based control. Computers and Chemical Engineering 32, 4 (2008), 754--765. Festschrift devoted to Rex Reklaitis on his 65th Birthday.
[17]
Liang Feng, Yuxiao Huang, Lei Zhou, Jinghui Zhong, Abhishek Gupta, Ke Tang, and Kay Chen Tan. 2021. Explicit Evolutionary Multitasking for Combinatorial Optimization: A Case Study on Capacitated Vehicle Routing Problem. IEEE Transactions on Cybernetics 51, 6 (2021), 3143--3156.
[18]
Anthony V. Fiacco. 1976. Sensitivity Analysis for Nonlinear Programming Using Penalty Methods. Math. Program. 10, 1 (dec 1976), 287--311.
[19]
Roger Fletcher. 1987. Practical Methods of Optimization (second ed.). John Wiley & Sons, New York, NY, USA.
[20]
Matthew C. Fontaine and Stefanos Nikolaidis. 2021. Differentiable Quality Diversity. In Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (Eds.), Vol. 34. Curran Associates, Inc., 10040--10052. https://proceedings.neurips.cc/paper_files/paper/2021/file/532923f11ac97d3e7cb0130315b067dc-Paper.pdf
[21]
Matthew C. Fontaine and Stefanos Nikolaidis. 2023. Covariance Matrix Adaptation MAP-Annealing. In Proceedings of the Genetic and Evolutionary Computation Conference (Lisbon, Portugal) (GECCO '23). Association for Computing Machinery, New York, NY, USA, 456--465.
[22]
Matthew C. Fontaine, Julian Togelius, Stefanos Nikolaidis, and Amy K. Hoover. 2020. Covariance Matrix Adaptation for the Rapid Illumination of Behavior Space. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference (Cancún, Mexico) (GECCO '20). Association for Computing Machinery, New York, NY, USA, 94--102.
[23]
Vincent François-Lavet, Peter Henderson, Riashat Islam, Marc G. Bellemare, and Joelle Pineau. 2018. An Introduction to Deep Reinforcement Learning. Foundations and Trends® in Machine Learning 11, 3-4 (2018), 219--354.
[24]
Tomas Gal and Josef Nedoma. 1972. Multiparametric linear programming. Management Science 18, 7 (1972), 406--422.
[25]
Daniele Gravina, Ahmed Khalifa, Antonios Liapis, Julian Togelius, and Georgios N. Yannakakis. 2019. Procedural Content Generation through Quality Diversity. 2019 IEEE Conference on Games (CoG) (2019), 1--8. https://api.semanticscholar.org/CorpusID:195848208
[26]
Yang Guan, Yangang Ren, Shengbo Eben Li, Qi Sun, Laiquan Luo, and Keqiang Li. 2020. Centralized Cooperation for Connected and Automated Vehicles at Intersections by Proximal Policy Optimization. IEEE Transactions on Vehicular Technology 69, 11 (2020), 12597--12608.
[27]
Abhishek Gupta and Yew-Soon Ong. 2018. Memetic Computation: The Mainspring of Knowledge Transfer in a Data-Driven Optimization Era.
[28]
Abhishek Gupta, Yew-Soon Ong, and Liang Feng. 2016. Multifactorial Evolution: Toward Evolutionary Multitasking. IEEE Transactions on Evolutionary Computation 20, 3 (June 2016), 343--357. Conference Name: IEEE Transactions on Evolutionary Computation.
[29]
Abhishek Gupta, Lei Zhou, Yew-Soon Ong, Zefeng Chen, and Yaqing Hou. 2022. Half a Dozen Real-World Applications of Evolutionary Multitasking, and More. IEEE Computational Intelligence Magazine 17 (05 2022), 49--66.
[30]
Nikolaus Hansen, Sibylle Müller, and Petros Koumoutsakos. 2003. Reducing the Time Complexity of the Derandomized Evolution Strategy with Covariance Matrix Adaptation (CMA-ES). Evolutionary computation 11 (02 2003), 1--18.
[31]
Xingxing Hao, Rong Qu, and Jing Liu. 2021. A Unified Framework of Graph-Based Evolutionary Multitasking Hyper-Heuristic. IEEE Transactions on Evolutionary Computation 25, 1 (2021), 35--47.
[32]
Shijia Huang, Jinghui Zhong, and Wei-Jie Yu. 2021. Surrogate-Assisted Evolutionary Framework with Adaptive Knowledge Transfer for Multi-Task Optimization. IEEE Transactions on Emerging Topics in Computing 9, 4 (2021), 1930--1944.
[33]
Binh Huynh Thi Thanh, Le Van Cuong, Ta Bao Thang, and Nguyen Hoang Long. 2023. Ensemble Multifactorial Evolution With Biased Skill-Factor Inheritance for Many-Task Optimization. IEEE Transactions on Evolutionary Computation 27, 6 (2023), 1735--1749.
[34]
Jeppe Theiss Kristensen and Paolo Burelli. 2020. Strategies for Using Proximal Policy Optimization in Mobile Puzzle Games. In Proceedings of the 15th International Conference on the Foundations of Digital Games (Bugibba, Malta) (FDG '20). Association for Computing Machinery, New York, NY, USA, Article 2, 10 pages.
[35]
Joel Lehman and Kenneth Stanley. 2011. Abandoning Objectives: Evolution Through the Search for Novelty Alone. Evolutionary computation 19 (06 2011), 189--223.
[36]
Jing Liang, Kangjia Qiao, Minghua Yuan, Kunjie Yu, Boyang Qu, Shilei Ge, Yaxin Li, and Guanlin Chen. 2020. Evolutionary multi-task optimization for parameters extraction of photovoltaic models. Energy Conversion and Management 207 (03 2020), 112509.
[37]
Zhengping Liang, Xiuju Xu, Ling Liu, Yaofeng Tu, and Zexuan Zhu. 2022. Evolutionary Many-Task Optimization Based on Multisource Knowledge Transfer. IEEE Transactions on Evolutionary Computation 26, 2 (2022), 319--333.
[38]
Rung-Tzuo Liaw and Chuan-Kang Ting. 2017. Evolutionary many-tasking based on biocoenosis through symbiosis: A framework and benchmark problems. In 2017 IEEE Congress on Evolutionary Computation (CEC). 2266--2273.
[39]
Siyu Lin and Peter A Beling. 2021. An end-to-end optimal trade execution framework based on proximal policy optimization. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence. 4548--4554.
[40]
Junwei Liu, Peiling Li, Guibin Wang, Yongxing Zha, Jianchun Peng, and Gang Xu. 2020. A Multitasking Electric Power Dispatch Approach With Multi-Objective Multifactorial Optimization Algorithm. IEEE Access 8 (2020), 155902--155911.
[41]
Songrit Maneewongvatana and David M Mount. 1999. Analysis of approximate nearest neighbor searching with clustered point sets. arXiv preprint cs/9901013 (1999).
[42]
Alan Tan Wei Min, Yew-Soon Ong, Abhishek Gupta, and Chi-Keong Goh. 2019. Multiproblem Surrogates: Transfer Evolutionary Multiobjective Optimization of Computationally Expensive Problems. IEEE Transactions on Evolutionary Computation 23, 1 (2019), 15--28.
[43]
Jean-Baptiste Mouret. 2023. Fast generation of centroids for MAP-Elites. In Companion Proceedings of the Conference on Genetic and Evolutionary Computation, GECCO 2023, Companion Volume, Lisbon, Portugal, July 15-19, 2023, Sara Silva and Luís Paquete (Eds.). ACM, 155--158.
[44]
Jean-Baptiste Mouret and Jeff Clune. 2015. Illuminating search spaces by mapping elites.
[45]
Jean-Baptiste Mouret and Glenn Maguire. 2020. Quality Diversity for Multi-Task Optimization. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference (Cancún, Mexico) (GECCO '20). Association for Computing Machinery, New York, NY, USA, 121--129.
[46]
Olle Nilsson and Antoine Cully. 2021. Policy gradient assisted MAP-Elites. In Proceedings of the Genetic and Evolutionary Computation Conference. ACM, Lille France, 866--875.
[47]
Michael Pearce and Juergen Branke. 2018. Continuous multi-task Bayesian Optimisation with correlation. European Journal of Operational Research 270, 3 (2018), 1074--1085.
[48]
Luca Pinciroli, Piero Baraldi, Guido Ballabio, Michele Compare, and Enrico Zio. 2021. Deep Reinforcement Learning Based on Proximal Policy Optimization for the Maintenance of a Wind Farm with Multiple Crews. Energies 14, 20 (2021).
[49]
Efstratios N. Pistikopoulos, Vivek Dua, Nikolaos A. Bozinis, Alberto Bemporad, and Manfred Morari. 2000. On-line optimization via off-line parametric optimization tools. Computers & Chemical Engineering 24, 2 (July 2000), 183--188.
[50]
Efstratios N Pistikopoulos, Vivek Dua, Nikolaos A Bozinis, Alberto Bemporad, and Manfred Morari. 2002. On-line optimization via off-line parametric optimization tools. Computers & Chemical Engineering 26, 2 (2002), 175--185.
[51]
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. 2021. Stable-Baselines3: Reliable Reinforcement Learning Implementations. Journal of Machine Learning Research 22, 268 (2021), 1--8. http://jmlr.org/papers/v22/20-1364.html
[52]
Ramon Sagarna and Yew-Soon Ong. 2016. Concurrently searching branches in software tests generation through multitask evolution. In 2016 IEEE Symposium Series on Computational Intelligence (SSCI). 1--8.
[53]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal Policy Optimization Algorithms. CoRR abs/1707.06347 (2017).
[54]
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando de Freitas. 2016. Taking the Human Out of the Loop: A Review of Bayesian Optimization. Proc. IEEE 104, 1 (2016), 148--175.
[55]
Haoran Sun, Linhan Yang, Yuping Gu, Jia Pan, Fang Wan, and Chaoyang Song. 2023. Bridging Locomotion and Manipulation Using Reconfigurable Robotic Limbs via Reinforcement Learning. Biomimetics 8, 4 (2023).
[56]
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. 2020. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods 17 (2020), 261--272.
[57]
Chao Wang, Jing Liu, Kai Wu, and Zhaoyang Wu. 2022. Solving Multitask Optimization Problems With Adaptive Knowledge Transfer via Anomaly Detection. IEEE Transactions on Evolutionary Computation 26, 2 (2022), 304--318.
[58]
Jian Yin, Anmin Zhu, Zexuan Zhu, Yanan Yu, and Xiaoliang Ma. 2019. Multifactorial Evolutionary Algorithm Enhanced with Cross-task Search Direction. In 2019 IEEE Congress on Evolutionary Computation (CEC). 2244--2251.
[59]
Gen Yokoya, Heng Xiao, and Toshiharu Hatanaka. 2019. Multifactorial optimization using Artificial Bee Colony and its application to Car Structure Design Optimization. In 2019 IEEE Congress on Evolutionary Computation (CEC). 3404--3409.
[60]
Ming Zhang, Yang Lu, Youxi Hu, Nasser Amaitik, and Yuchun Xu. 2022. Dynamic Scheduling Method for Job-Shop Manufacturing Systems by Deep Reinforcement Learning with Proximal Policy Optimization. Sustainability 14, 9 (2022).
[61]
Hong Zhao, Xuhui Ning, Xiaotao Liu, Chao Wang, and Jing Liu. 2023. What makes evolutionary multi-task optimization better: A comprehensive survey. Applied Soft Computing 145 (2023), 110545.
[62]
Jinghui Zhong, Liang Feng, Wentong Cai, and Yew-Soon Ong. 2020. Multifactorial Genetic Programming for Symbolic Regression Problems. IEEE Transactions on Systems, Man, and Cybernetics: Systems 50, 11 (2020), 4492-4505.
[63]
Jacques Zhong, Vincent Weistroffer, Jean-Baptiste Mouret, Francis Colas, and Pauline Maurice. 2023. Workstation Suitability Maps: Generating Ergonomic Behaviors on a Population of Virtual Humans With Multi-Task Optimization. IEEE Robotics Autom. Lett. 8, 11 (2023), 7384--7391.
Index Terms
- Parametric-Task MAP-Elites
Index terms have been assigned to the content through auto-classification.
Recommendations
Multi-Task Multi-Behavior MAP-Elites
GECCO '23 Companion: Proceedings of the Companion Conference on Genetic and Evolutionary ComputationWe propose Multi-Task Multi-Behavior MAP-Elites, a variant of MAP-Elites that finds a large number of high-quality solutions for a large set of tasks (optimization problems from a given family). It combines the original MAP-Elites for the search for ...
MAP-Elites with Descriptor-Conditioned Gradients and Archive Distillation into a Single Policy
GECCO '23: Proceedings of the Genetic and Evolutionary Computation ConferenceQuality-Diversity algorithms, such as MAP-Elites, are a branch of Evolutionary Computation generating collections of diverse and high-performing solutions, that have been successfully applied to a variety of domains and particularly in evolutionary ...
Blending notions of diversity for MAP-elites
GECCO '19: Proceedings of the Genetic and Evolutionary Computation Conference CompanionQuality-diversity algorithms focus on discovering multiple diverse and high-performing solutions. MAP-elites is such an algorithm, as it partitions the solution space into bins and searches for the best solution possible for each bin. In this paper, ...
Comments
Information & Contributors
Information
Published In
July 2024
1657 pages
Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 14 July 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- ANR
- Horizon Europe
- Agence de l'Innovation de Défense
Conference
GECCO '24
Sponsor:
GECCO '24: Genetic and Evolutionary Computation Conference
July 14 - 18, 2024
VIC, Melbourne, Australia
Acceptance Rates
Overall Acceptance Rate 1,669 of 4,410 submissions, 38%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 16Total Downloads
- Downloads (Last 12 months)16
- Downloads (Last 6 weeks)6
Reflects downloads up to 03 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in