research-article

Bayesian Optimisation for Constrained Problems

Authors:

Juergen BrankeAuthors Info & Claims

ACM Transactions on Modeling and Computer Simulation, Volume 34, Issue 2

Article No.: 9, Pages 1 - 26

https://doi.org/10.1145/3641544

Published: 08 April 2024 Publication History

Abstract

Many real-world optimisation problems such as hyperparameter tuning in machine learning or simulation-based optimisation can be formulated as expensive-to-evaluate black-box functions. A popular approach to tackle such problems is Bayesian optimisation, which builds a response surface model based on the data collected so far, and uses the mean and uncertainty predicted by the model to decide what information to collect next. In this article, we propose a generalisation of the well-known Knowledge Gradient acquisition function that allows it to handle constraints. We empirically compare the new algorithm with four other state-of-the-art constrained Bayesian optimisation algorithms and demonstrate its superior performance. We also prove theoretical convergence in the infinite budget limit.

References

[1]

Mauricio A. Álvarez, Lorenzo Rosasco, and Neil D. Lawrence. 2012. Kernels for vector-valued functions: A review. Foundations and Trends® in Machine Learning 4, 3 (March 2012), 195–266. DOI:

Digital Library

[2]

Satyajith Amaran, Nikolaos V. Sahinidis, Bikram Sharda, and Scott J. Bury. 2016. Simulation optimization: A review of algorithms and applications. Annals of Operations Research 240, 1 (2016), 351–380.

[3]

Samineh Bagheri, Wolfgang Konen, Richard Allmendinger, Juergen Branke, Kalyanmoy Deb, Jonathan Fieldsend, Domenico Quagliarella, and Karthik Sindhya. 2017. Constraint handing in efficient global optimization. In Proceedings of the Genetic and Evolutionary Computation Conference. ACM, 673–680.

Digital Library

[4]

Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, and Eytan Bakshy. 2020. BoTorch: A framework for efficient Monte-Carlo Bayesian optimization. In Advances in Neural Information Processing Systems 33. http://arxiv.org/abs/1910.06403

[5]

Felix Berkenkamp, Andreas Krause, and Angela P. Schoellig. 2016. Bayesian optimization with safety constraints: Safe and automatic parameter tuning in robotics. arXiv abs/1602.04450 (2016).

[6]

Felix Berkenkamp, Andreas Krause, and Angela P. Schoellig. 2023. Bayesian optimization with safety constraints: Safe and automatic parameter tuning in robotics. Machine Learning 112 (2023), 3713–3747.

[7]

Antonio Candelieri. 2019. Sequential model based optimization of partially defined functions under unknown constraints. Journal of Global Optimization 79 (2019), 281–303.

[8]

Wenjie Chen, Shengcai Liu, and Ke Tang. 2021. A new knowledge gradient-based method for constrained Bayesian optimization. arXiv:2101.08743 [cs.LG] (2021).

[9]

Erhan Cinlar. 2011. Probability and Stochastics. Graduate Texts in Mathematics, Vol. 261. Springer.

[10]

David Eriksson and Matthias Poloczek. 2021. Scalable constrained Bayesian optimization. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics, Arindam Banerjee and Kenji Fukumizu (Eds.). Proceedings of Machine Learning Research, Vol. 130. PMLR, 730–738. https://proceedings.mlr.press/v130/eriksson21a.html

[11]

Alexander I. J. Forrester, Andras Sobester, and Andy J. Keane. 2008. Engineering Design via Surrogate Modelling. Wiley.

[12]

Peter I. Frazier. 2018. A tutorial on Bayesian optimization. arXiv:1807.02811 [stat.ML] (2008).

[13]

Jacob Gardner, Matt Kusner, Eddie Xu, Kilian Weinberger, and John Cunningham. 2014. Bayesian optimization with inequality constraints. In Proceedings of the 31st International Conference on Machine Learning (ICML’14). 3.

[14]

Robert B. Gramacy, Genetha A. Gray, Sébastien Le Digabel, Herbert K. H. Lee, Pritam Ranjan, Garth Wells, and Stefan M. Wild. 2016. Modeling an augmented Lagrangian for blackbox constrained optimization. Technometrics 58, 1 (2016), 1–11. DOI:

[15]

José Miguel Hernández-Lobato, Michael A. Gelbart, Ryan P. Adams, Matthew W. Hoffman, and Zoubin Ghahramani. 2016. A general framework for constrained Bayesian optimization using information-based search. Journal of Machine Learning Research 17, 1 (Jan.2016), 5549–5601.

Digital Library

[16]

Jose Miguel Hernandez-Lobato, Matthew W. Hoffman, and Zoubin Ghahramani. 2014. Predictive entropy search for efficient global optimization of black-box functions. In Proceedings of the 27th International Conference on Neural Information Processing Systems—Volume 1 (NIPS’14). MIT Press, Cambridge, MA, 918–926.

[17]

Donald R. Jones, Matthias Schonlau, and William J. Welch. 1998. Efficient global optimization of expensive black-box functions. Journal of Global Optimization 13 (Jan.1998), 455–492. DOI:

Digital Library

[18]

Kristian Kersting, Christian Plagemann, Patrick Pfaff, and Wolfram Burgard. 2007. Most likely heteroscedastic Gaussian process regression. In Proceedings of the 24th International Conference on Machine Learning(ICML’07). ACM, 393–400. DOI:

Digital Library

[19]

Jack P. C. Kleijnen, Inneke Van Nieuwenhuyse, and Wim van Beers. 2021. Constrained Optimization in Simulation: Efficient Global Optimization and Karush-Kuhn-Tucker Conditions. CentER Discussion Paper Series No. 2021-031. CentER, Cenber for Economic Research. DOI:

[20]

Remi R. Lam and Karen E. Willcox. 2017. Lookahead Bayesian optimization with inequality constraints. In Proceedings of the 31st International Conference on Neural Information Processing Systems(NIPS’17). 1888–1898.

Digital Library

[21]

Benjamin Letham, Brian Karrer, Guilherme Ottoni, and Eytan Bakshy. 2017. Constrained Bayesian optimization with noisy experiments. Bayesian Analysis 14, 2 (2017), 495–519. DOI:

[22]

Michael Pearce and Juergen Branke. 2018. Continuous multi-task Bayesian optimisation with correlation. European Journal of Operational Research 270, 3 (2018), 1074–1085.

[23]

Michael Pearce, Janis Klaise, and Matthew Groves. 2020. Practical Bayesian optimization of objectives with conditioning variables. arXiv:2002.09996 [stat.ML] (2020).

[24]

Michael Pearce, Matthias Poloczek, and Juergen Branke. 2019. Bayesian simulation optimization with common random numbers. In Proceedings of the Winter Simulation Conference(WSC’19). IEEE, 3492–3503.

Digital Library

[25]

Julien Pelamatti, Rodolphe Le Riche, Céline Helbert, and Christophette Blanchet-Scalliet. 2022. Coupling and selecting constraints in Bayesian optimization under uncertainties. arXiv:2204.00527 (2022). DOI:

[26]

Victor Picheny. 2014. A stepwise uncertainty reduction approach to constrained global optimization. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, Samuel Kaski and Jukka Corander (Eds.). Proceedings of Machine Learning Research, Vol. 33. PMLR, 787–795. http://proceedings.mlr.press/v33/picheny14.html

[27]

Victor Picheny, Robert B. Gramacy, Stefan Wild, and Sebastien Le Digabel. 2016. Bayesian optimization under mixed constraints with a slack-variable augmented Lagrangian. In Advances in Neural Information Processing Systems 29, D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (Eds.). Curran Associates, 1435–1443. http://papers.nips.cc/paper/6439-bayesian-optimization-under-mixed-constraints-with-a-slack-variable-augmented-lagrangian.pdf

[28]

Victor Picheny, Tobias Wagner, and David Ginsbourger. 2013. A benchmark of kriging-based infill criteria for noisy optimization. Structural and Multidisciplinary Optimization 48 (2013), 607–626. DOI:

Digital Library

[29]

Matthias Poloczek, Jialei Wang, and Peter Frazier. 2017. Multi-information source optimization. In Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, 1–11.https://proceedings.neurips.cc/paper/2017/file/df1f1d20ee86704251795841e6a9405a-Paper.pdf

[30]

Carl E. Rasmussen and Christopher K. I. Williams. 2006. Gaussian Processes for Machine Learning. MIT Press, Cambridge, MA.

Digital Library

[31]

Michael Sasena. 2002. Flexibility and Efficiency Enhancements for Constrained Global Design Optimization with Kriging Approximations. Ph. D. Dissertation. University of Michigan.

[32]

Matthias Schonlau, William Welch, and Donald Jones. 1998. Global versus local search in constrained optimization of computer models. IMS Lecture Notes Monogram Series 34 (1998), 11–25. DOI:

[33]

Warren Scott, Peter Frazier, and Warren Powell. 2011. The correlated knowledge gradient for simulation optimization of continuous parameters using Gaussian process regression. SIAM Journal on Optimization 21, 3 (2011), 996–1026. DOI:

[34]

Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando de Freitas. 2016. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE 104, 1 (2016), 148–175.

[35]

Jian Wu and Peter I. Frazier. 2017. Discretization-free knowledge gradient methods for Bayesian optimization. arXiv:1707.06541 [stat.ML] (2017).

[36]

Yunxiang Zhang, Xiangyu Zhang, and Peter Frazier. 2021. Constrained two-step look-ahead Bayesian optimization. In Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS’21). 1–13.

Cited By

Edmunds EThomas M(2025)Classification of microstructural defects in selective laser melted inconel 713C alloy using convolutional neural networksMaterials Science and Technology10.1177/02670836241308470Online publication date: 15-Jan-2025
https://doi.org/10.1177/02670836241308470
Amini SVannieuwenhuyse IMorales-Hernández A(2025)Constrained Bayesian Optimization: A ReviewIEEE Access10.1109/ACCESS.2024.352287613(1581-1593)Online publication date: 2025
https://doi.org/10.1109/ACCESS.2024.3522876
Nair AArivazhagan M(2024)Industrial activated sludge model identification using hyperparameter-tuned metaheuristicsSwarm and Evolutionary Computation10.1016/j.swevo.2024.10173391(101733)Online publication date: Dec-2024
https://doi.org/10.1016/j.swevo.2024.101733

Index Terms

Bayesian Optimisation for Constrained Problems

Recommendations

Bayesian Optimisation vs. Input Uncertainty Reduction
Simulators often require calibration inputs estimated from real-world data, and the estimate can significantly affect simulation output. Particularly when performing simulation optimisation to find an optimal solution, the uncertainty in the inputs ...
How Bayesian should Bayesian optimisation be?
GECCO '21: Proceedings of the Genetic and Evolutionary Computation Conference Companion

Bayesian optimisation (BO) uses probabilistic surrogate models - usually Gaussian processes (GPs) - for the optimisation of expensive black-box functions. At each BO iteration, the GP hyperparameters are fit to previously-evaluated data by maximising ...
Are we forgetting about compositional optimisers in Bayesian optimisation?

Bayesian optimisation presents a sample-efficient methodology for global optimisation. Within this framework, a crucial performance-determining subroutine is the maximisation of the acquisition function, a task complicated by the fact that acquisition ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Modeling and Computer Simulation

ACM Transactions on Modeling and Computer Simulation Volume 34, Issue 2

April 2024

178 pages

EISSN:1558-1195

DOI:10.1145/3613554

Editor:
Wentong Cai
Nanyang Technological University, Singapore

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 April 2024

Online AM: 22 January 2024

Accepted: 31 December 2023

Revised: 01 August 2023

Received: 14 January 2022

Published in TOMACS Volume 34, Issue 2

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
615
Total Downloads

Downloads (Last 12 months)601
Downloads (Last 6 weeks)27

Reflects downloads up to 12 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Edmunds EThomas M(2025)Classification of microstructural defects in selective laser melted inconel 713C alloy using convolutional neural networksMaterials Science and Technology10.1177/02670836241308470Online publication date: 15-Jan-2025
https://doi.org/10.1177/02670836241308470
Amini SVannieuwenhuyse IMorales-Hernández A(2025)Constrained Bayesian Optimization: A ReviewIEEE Access10.1109/ACCESS.2024.352287613(1581-1593)Online publication date: 2025
https://doi.org/10.1109/ACCESS.2024.3522876
Nair AArivazhagan M(2024)Industrial activated sludge model identification using hyperparameter-tuned metaheuristicsSwarm and Evolutionary Computation10.1016/j.swevo.2024.10173391(101733)Online publication date: Dec-2024
https://doi.org/10.1016/j.swevo.2024.101733

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

Figures

Tables

Media

View full text|Download PDF

View Issue’s Table of Contents