Abstract
Game theory finds nowadays a broad range of applications in engineering and machine learning. However, in a derivative-free, expensive black-box context, very few algorithmic solutions are available to find game equilibria. Here, we propose a novel Gaussian-process based approach for solving games in this context. We follow a classical Bayesian optimization framework, with sequential sampling decisions based on acquisition functions. Two strategies are proposed, based either on the probability of achieving equilibrium or on the stepwise uncertainty reduction paradigm. Practical and numerical aspects are discussed in order to enhance the scalability and reduce computation time. Our approach is evaluated on several synthetic game problems with varying number of players and decision space dimensions. We show that equilibria can be found reliably for a fraction of the cost (in terms of black-box evaluations) compared to classical, derivative-based algorithms. The method is available in the R package GPGame available on CRAN at https://cran.r-project.org/package=GPGame.
Similar content being viewed by others
References
Adams, R.A., Fournier, J.J.: Sobolev Spaces, vol. 140. Academic Press, Cambridge (2003)
Álvarez, M.A., Rosasco, L., Lawrence, N.D.: Kernels for vector-valued functions: a review. Found. Trend Mach. Learn. 4(3), 195–266 (2011). https://doi.org/10.1561/2200000036
Azzalini, A., Genz, A.: The R package mnormt: the multivariate normal and \(t\) distributions (version 1.5–4). http://azzalini.stat.unipd.it/SW/Pkg-mnormt (2016). Accessed 8 Mar 2016
Başar, T.: Relaxation techniques and asynchronous algorithms for on-line computation of noncooperative equilibria. J. Econ. Dyn. Control. 11(4), 531–549 (1987)
Bect, J., Ginsbourger, D., Li, L., Picheny, V., Vazquez, E.: Sequential design of computer experiments for the estimation of a probability of failure. Stat. Comput. 22(3), 773–793 (2012)
Bect, J., Bachoc, F., Ginsbourger, D.: A supermartingale approach to Gaussian process based sequential design of experiments. arXiv preprint arXiv:1608.01118 (2016)
Brown, N., Ganzfried, S., Sandholm, T.: Hierarchical abstraction, distributed equilibrium computation, and post-processing, with application to a champion no-limit Texas hold’em agent. In: Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, pp. 7–15 (2015)
Chevalier, C., Ginsbourger, D.: Fast computation of the multi-points expected improvement with applications in batch selection. In: Learning and Intelligent Optimization, Springer, pp. 59–69 (2013)
Chevalier, C., Emery, X., Ginsbourger, D.: Fast update of conditional simulation ensembles. Math. Geosci. 47(7), 771–789 (2015)
Cressie, N.: Statistics for spatial data. Terra Nova 4(5), 613–617 (1992)
Dorsch, D., Jongen, H.T., Shikhman, V.: On structure and computation of generalized nash equilibria. SIAM J. Optim. 23(1), 452–474 (2013)
Facchinei, F., Kanzow, C.: Generalized nash equilibrium problems. Annal. Oper. Res. 175(1), 177–211 (2010)
Fleuret, F., Geman, D.: Graded learning for object detection. In: Proceedings of the Workshop on Statistical and Computational Theories of Vision of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR/SCTV), vol. 2 (1999)
Friedman, A.: Stochastic differential games. J. Differ. Equ. 11(1), 79–108 (1972)
Games, I.L.S.C.: Lenient learning in independent-learner stochastic cooperative games. J. Mach. Learn. Res. 17, 1–42 (2016)
Garivier, A., Kaufmann, E., Koolen, W. M.: Maximin action identification: a new bandit framework for games. In: 29th Annual Conference on Learning Theory, pp. 1028–1050 (2016)
Genz, A., Bretz, F.: Computation of Multivariate Normal and t Probabilities. Lecture Notes in Statistics. Springer, Heidelberg (2009)
Genz, A., Bretz, F., Miwa, T., Mi, X., Leisch, F., Scheipl, F., Hothorn, T.: mvtnorm: Multivariate normal and t Distributions. http://CRAN.R-project.org/package=mvtnorm, r package version 1.0–5 (2016). Accessed 2 Feb 2016
Gibbons, R.: Game Theory for Applied Economists. Princeton University Press, Princeton (1992)
Ginsbourger, D., Le Riche, R.: Towards Gaussian process-based optimization with finite time horizon. In: mODa9–Advances in Model-Oriented Design and Analysis, Springer, pp. 89–96 (2010)
Gonzalez, J., Osborne, M., Lawrence, N.: Glasses: relieving the myopia of Bayesian optimisation. In: Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, pp. 790–799 (2016)
Gramacy, R.B., Apley, D.W.: Local gaussian process approximation for large computer experiments. J. Comput. Graph. Stat. 24(2), 561–578 (2015)
Gramacy, R.B., Ludkovski, M.: Sequential design for optimal stopping problems. SIAM J. Financ. Math. 6(1), 748–775 (2015)
Habbal, A., Kallel, M.: Neumann–Dirichlet Nash strategies for the solution of elliptic Cauchy problems. SIAM J. Control Optim. 51(5), 4066–4083 (2013). https://doi.org/10.1137/120869808
Habbal, A., Petersson, J., Thellner, M.: Multidisciplinary topology optimization solved as a Nash game. Int. J. Numer. Methods Eng. 61, 949–963 (2004)
Harsanyi, J.C.: Games with randomly disturbed payoffs: a new rationale for mixed-strategy equilibrium points. Int. J. Game Theory 2(1), 1–23 (1973)
Heaton, M. J., Datta, A., Finley, A., Furrer, R., Guhaniyogi, R., Gerber, F., Gramacy, R. B., Hammerling, D., Katzfuss, M., Lindgren, F., et al.: A case study competition among methods for analyzing large spatial data. arXiv preprint arXiv:1710.05013 (2017)
Hecht, F., Pironneau, O., Le Hyaric, A., Ohtsuka, K.: Freefem++ v. 2.11. Users? Manual University of Paris 6 (2010)
Hennig, P., Schuler, C.J.: Entropy search for information-efficient global optimization. J. Mach. Learn. Res. 13, 1809–1837 (2012)
Hernández-Lobato, J.M., Hoffman, M.W., Ghahramani, Z.: Predictive entropy search for efficient global optimization of black-box functions. In: Advances in neural information processing systems, pp. 918–926 (2014)
Hernández-Lobato, J.M., Gelbart, M.A., Adams, R.P., Hoffman, M.W., Ghahramani, Z.: A general framework for constrained bayesian optimization using information-based search. J. Mach. Learn. Res. 17(160), 1–53 (2016)
Hu, J., Wellman, M.P.: Nash q-learning for general-sum stochastic games. J. Mach. Learn. Res. 4, 1039–1069 (2003)
Isaacs, R.: Differential Games. A Mathematical Theory with Applications to Warfare and Pursuit, Control and Optimization. Wiley, New York (1965)
Jala, M., Lévy-Leduc, C., Moulines, É., Conil, E., Wiart, J.: Sequential design of computer experiments for the assessment of fetus exposure to electromagnetic fields. Technometrics 58(1), 30–42 (2016)
Johanson, M., Bowling, M.H.: Data biased robust counter strategies. In: Proceedings of the Twelfth International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 264–271 (2009)
Jones, D.R., Schonlau, M., Welch, W.J.: Efficient global optimization of expensive black-box functions. J. Glob. Optim. 13(4), 455–492 (1998)
Kanzow, C., Steck, D.: Augmented lagrangian methods for the solution of generalized nash equilibrium problems. SIAM J. Optim. 26(4), 2034–2058 (2016)
Lanctot, M., Burch, N., Zinkevich, M., Bowling, M., Gibson, R.G.: No-regret learning in extensive-form games with imperfect recall. In: Proceedings of the 29th International Conference on Machine Learning (ICML-12), pp. 65–72 (2012)
León, E.R., Pape, A.L., Désidéri, J.A., Alfano, D., Costes, M.: Concurrent aerodynamic optimization of rotor blades using a nash game method. J. Am. Helicopter Soc. 61, 1–13 (2014)
Li, S., Başar, T.: Distributed algorithms for the computation of noncooperative equilibria. Autom. J. IFAC 23(4), 523–533 (1987)
Littman, M.L., Stone, P.: A polynomial-time nash equilibrium algorithm for repeated games. Decis. Support Syst. 39(1), 55–66 (2005)
McKay, M.D., Beckman, R.J., Conover, W.J.: Comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21(2), 239–245 (1979)
Mockus, J.: Bayesian Approach to Global Optimization: Theory and Applications. Springer, Berlin (1989)
Neyman, A., Sorin, S.: Stochastic Games and Applications, vol. 570. Springer, Berlin (2003)
Nishimura, R., Hayashi, S., Fukushima, M.: Robust nash equilibria in n-person non-cooperative games: uniqueness and reformulation. Pac. J. Optim. 5(2), 237–259 (2009)
Parr, J. M.: Improvement Criteria for Constraint Handling and Multiobjective Optimization. Ph.D thesis, University of Southampton (2012)
Picheny, V.: A stepwise uncertainty reduction approach to constrained global optimization. In: Proceedings of the 17th International Conference on Artificial Intelligence and Statistics, JMLR W&CP, vol 33, pp. 787–795 (2014)
Picheny, V., Binois, M.: GPGame: solving complex game problems using Gaussian processes. URL http://CRAN.R-project.org/package=GPGame, r package version 0.1.3 (2017)
Plumlee, M.: Fast prediction of deterministic functions using sparse grid experimental designs. J. Am. Stat. Assoc. 109(508), 1581–1591 (2014)
R Core Team (2016) R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna. https://www.R-project.org/. Accessed 15 Mar 2018
Rasmussen, C.E., Williams, C.: Gaussian Processes for Machine Learning. MIT Press. http://www.gaussianprocess.org/gpml/ (2006)
Rosenmüller, J.: On a generalization of the lemke-howson algorithm to noncooperative n-person games. SIAM J. Appl. Math. 21(1), 73–79 (1971)
Roustant, O., Ginsbourger, D., Deville, Y.: DiceKriging, DiceOptim: two R packages for the analysis of computer experiments by kriging-based metamodeling and optimization. J. Stat. Softw. 51(1), 1–55 (2012)
Rullière, D., Durrande, N., Bachoc, F., Chevalier, C.: Nested kriging predictions for datasets with a large number of observations. Stat. Comput. 28, 1–19 (2016)
Scilab Enterprises (2012) Scilab: Free and Open Source Software for Numerical Computation. Scilab Enterprises, Orsay. http://www.scilab.org. Accessed 1 Apr 2015
Shahriari, B., Swersky, K., Wang, Z., Adams, R.P., de Freitas, N.: Taking the human out of the loop: a review of bayesian optimization. Proc. IEEE 104(1), 148–175 (2016)
Shapley, L.S.: Stochastic games. Proc. Natl. Acad. Sci. 39(10), 1095–1100 (1953)
Srinivas, N., Krause, A., Kakade, S.M., Seeger, M.: Information-theoretic regret bounds for gaussian process optimization in the bandit setting. Inf. Theory IEEE Trans. 58(5), 3250–3265 (2012)
Uryas’ev, S., Rubinstein, R.Y.: On relaxation algorithms in computation of noncooperative equilibria. IEEE Trans. Autom. Control 39(6), 1263–1267 (1994)
Villemonteix, J., Vazquez, E., Walter, E.: An informational approach to the global optimization of expensive-to-evaluate functions. J. Glob. Optim. 44(4), 509–534 (2009)
Wagner, T., Emmerich, M., Deutz, A., Ponweiser, W.: On expected-improvement criteria for model-based multi-objective optimization. In: International Conference on Parallel Problem Solving from Nature, Springer, Berlin. pp. 718–727 (2010)
Wang, G., Shan, S.: Review of metamodeling techniques in support of engineering design optimization. J. Mech. Des. 129(4), 370 (2007)
Wilson, A., Nickisch, H.: Kernel interpolation for scalable structured Gaussian processes (kiss-gp). In: International Conference on Machine Learning, pp. 1775–1784 (2015)
Žilinskas, A., Zhigljavsky, A.: Stochastic global optimization: a review on the occasion of 25 years of informatica. Informatica 27(2), 229–256 (2016)
Acknowledgements
The authors acknowledge inspiration from Lorentz Center Workshop “SAMCO-Surrogate Model Assisted Multicriteria Optimization”, at Leiden University Feb 29–March 4, 2016. Mickal Binois is grateful for support from National Science Foundation Grant DMS-1521702.
Author information
Authors and Affiliations
Corresponding author
Appendices
Handling conditional simulations
We detail here how we generate the draws of \(\mathbf {Y}| \varvec{\mathcal {F}}_i\) to compute \({\hat{J}}(\mathbf {x})\) in practice. We employ the FOXY (fast update of conditional simulation ensemble) algorithm proposed by Chevalier et al. [9], as detailed below.
Let \(\varvec{\mathcal {Y}}_1, \ldots , \varvec{\mathcal {Y}}_M\) be independent draws of \(\mathbf {Y}\left( \mathbb {X}\right) \) (each \(\varvec{\mathcal {Y}}_i \in \mathbb {R}^{N \times p}\)), generated using the posterior Gaussian distribution of Eq. (8), and \(\varvec{\mathcal {F}}_1, \ldots , \varvec{\mathcal {F}}_K\) independent (of each other and of the \(\varvec{\mathcal {Y}}_i\)’s) draws of \(\mathbf {Y}(\mathbf {x}) + \varvec{\varepsilon }\) from the posterior Gaussian distribution of Eq. (9). As shown in Chevalier et al. [9], draws of \(\mathbf {Y}| \varvec{\mathcal {F}}_i\) can be obtained efficiently from \(\varvec{\mathcal {Y}}_1, \ldots , \varvec{\mathcal {Y}}_M\) using:
with \(1 \le i \le p\), \(1 \le j \le M\), \(1 \le k \le K\) and
Notice that \(\varvec{\lambda }^{(i)}(\mathbf {x})\) may only be computed once for all \(\mathcal {Y}_j^{(i)}(\mathbf {x})\).
\(C(\mathbf {x})\) formulae
For a given target \(T_E \in \mathbb {R}^p\) and \(\mathbf {x}\in \mathbb {X}\):
with \(\phi \) the probability density function of the standard Gaussian variable.
Let \(T_L \in \mathbb {R}^p\) and \(T_U \in \mathbb {R}^p\) such that \(\forall 1 \le i \le p, T_{Li} < T_{Ui}\) define a box in the objective space. Defining \(\varvec{\varPsi } = \left[ \varPsi (\varvec{\mathcal {Y}}_1), \ldots , \varPsi (\varvec{\mathcal {Y}}_M) \right] \) the \(p \times M\) matrix of simulated NE, we use:
Then, the probability to belong to the box is:
Solving NEP on GP draws
We detail here a simple algorithm to extract Nash equilibria from GP draws.
Computational time
We report here the computational time required to perform a single iteration of our algorithm for each of the three examples (not including the time required to run the simulation itself). Experiments were run on an Intel®Core\(^{{\mathrm{TM}}}\) i7-5600U CPU at 2.60GHz with 4 \(\times \) 8GB of RAM.
Rights and permissions
About this article
Cite this article
Picheny, V., Binois, M. & Habbal, A. A Bayesian optimization approach to find Nash equilibria. J Glob Optim 73, 171–192 (2019). https://doi.org/10.1007/s10898-018-0688-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10898-018-0688-0