Showing 1–50 of 57 results for author: Laurière, M

Global Solutions to Master Equations for Continuous Time Heterogeneous Agent Macroeconomic Models

Authors: Zhouzhou Gu, Mathieu Laurière, Sebastian Merkel, Jonathan Payne

Abstract: We propose and compare new global solution algorithms for continuous time heterogeneous agent economies with aggregate shocks. First, we approximate the agent distribution so that equilibrium in the economy can be characterized by a high, but finite, dimensional non-linear partial differential equation. We consider different approximations: discretizing the number of agents, discretizing the agent… ▽ More We propose and compare new global solution algorithms for continuous time heterogeneous agent economies with aggregate shocks. First, we approximate the agent distribution so that equilibrium in the economy can be characterized by a high, but finite, dimensional non-linear partial differential equation. We consider different approximations: discretizing the number of agents, discretizing the agent state variables, and projecting the distribution onto a finite set of basis functions. Second, we represent the value function using a neural network and train it to solve the differential equation using deep learning tools. We refer to the solution as an Economic Model Informed Neural Network (EMINN). The main advantage of this technique is that it allows us to find global solutions to high dimensional, non-linear problems. We demonstrate our algorithm by solving important models in the macroeconomics and spatial literatures (e.g. Krusell and Smith (1998), Khan and Thomas (2007), Bilal (2023)). △ Less

Submitted 19 June, 2024; originally announced June 2024.

Machine Learning Methods for Large Population Games with Applications in Operations Research

Authors: Gokce Dayanikli, Mathieu Lauriere

Abstract: In this tutorial, we provide an introduction to machine learning methods for finding Nash equilibria in games with large number of agents. These types of problems are important for the operations research community because of their applicability to real life situations such as control of epidemics, optimal decisions in financial markets, electricity grid management, or traffic control for self-dri… ▽ More In this tutorial, we provide an introduction to machine learning methods for finding Nash equilibria in games with large number of agents. These types of problems are important for the operations research community because of their applicability to real life situations such as control of epidemics, optimal decisions in financial markets, electricity grid management, or traffic control for self-driving cars. We start the tutorial by introducing stochastic optimal control problems for a single agent, in discrete time and in continuous time. Then, we present the framework of dynamic games with finite number of agents. To tackle games with a very large number of agents, we discuss the paradigm of mean field games, which provides an efficient way to compute approximate Nash equilibria. Based on this approach, we discuss machine learning algorithms for such problems. First in the context of discrete time games, we introduce fixed point based methods and related methods based on reinforcement learning. Second, we discuss machine learning methods that are specific to continuous time problems, by building on optimality conditions phrased in terms of stochastic or partial differential equations. Several examples and numerical illustrations of problems arising in operations research are provided along the way. △ Less

Submitted 14 June, 2024; originally announced June 2024.

Comments: 39 pages, 11 figures

Analysis of Multiscale Reinforcement Q-Learning Algorithms for Mean Field Control Games

Authors: Andrea Angiuli, Jean-Pierre Fouque, Mathieu Laurière, Mengrui Zhang

Abstract: Mean Field Control Games (MFCG), introduced in [Angiuli et al., 2022a], represent competitive games between a large number of large collaborative groups of agents in the infinite limit of number and size of groups. In this paper, we prove the convergence of a three-timescale Reinforcement Q-Learning (RL) algorithm to solve MFCG in a model-free approach from the point of view of representative agen… ▽ More Mean Field Control Games (MFCG), introduced in [Angiuli et al., 2022a], represent competitive games between a large number of large collaborative groups of agents in the infinite limit of number and size of groups. In this paper, we prove the convergence of a three-timescale Reinforcement Q-Learning (RL) algorithm to solve MFCG in a model-free approach from the point of view of representative agents. Our analysis uses a Q-table for finite state and action spaces updated at each discrete time-step over an infinite horizon. In [Angiuli et al., 2023], we proved convergence of two-timescale algorithms for MFG and MFC separately highlighting the need to follow multiple population distributions in the MFC case. Here, we integrate this feature for MFCG as well as three rates of update decreasing to zero in the proper ratios. Our technique of proof uses a generalization to three timescales of the two-timescale analysis in [Borkar, 1997]. We give a simple example satisfying the various hypothesis made in the proof of convergence and illustrating the performance of the algorithm. △ Less

Submitted 3 June, 2024; v1 submitted 27 May, 2024; originally announced May 2024.

Comments: arXiv admin note: text overlap with arXiv:2312.06659

Learning equilibria in Cournot mean field games of controls

Authors: Fabio Camilli, Mathieu Laurière, Qing Tang

Abstract: We consider Cournot mean field games of controls, a model originally developed for the production of an exhaustible resource by a continuum of producers. We prove uniqueness of the solution under general assumptions on the price function. Then, we prove convergence of a learning algorithm which gives existence of a solution to the mean field games system. The learning algorithm is implemented with… ▽ More We consider Cournot mean field games of controls, a model originally developed for the production of an exhaustible resource by a continuum of producers. We prove uniqueness of the solution under general assumptions on the price function. Then, we prove convergence of a learning algorithm which gives existence of a solution to the mean field games system. The learning algorithm is implemented with a suitable finite difference discretization to get a numerical method to the solution. We supplement our theoretical analysis with several numerical examples and illustrate the impacts of model parameters. △ Less

Submitted 2 May, 2024; originally announced May 2024.

Deep Backward and Galerkin Methods for the Finite State Master Equation

Authors: Asaf Cohen, Mathieu Laurière, Ethan Zell

Abstract: This paper proposes and analyzes two neural network methods to solve the master equation for finite-state mean field games (MFGs). Solving MFGs provides approximate Nash equilibria for stochastic, differential games with finite but large populations of agents. The master equation is a partial differential equation (PDE) whose solution characterizes MFG equilibria for any possible initial distribut… ▽ More This paper proposes and analyzes two neural network methods to solve the master equation for finite-state mean field games (MFGs). Solving MFGs provides approximate Nash equilibria for stochastic, differential games with finite but large populations of agents. The master equation is a partial differential equation (PDE) whose solution characterizes MFG equilibria for any possible initial distribution. The first method we propose relies on backward induction in a time component while the second method directly tackles the PDE without discretizing time. For both approaches, we prove two types of results: there exist neural networks that make the algorithms' loss functions arbitrarily small, and conversely, if the losses are small, then the neural networks are good approximations of the master equation's solution. We conclude the paper with numerical experiments on benchmark problems from the literature up to dimension 15, and a comparison with solutions computed by a classical method for fixed initial distributions. △ Less

Submitted 7 March, 2024; originally announced March 2024.

A Deep Learning Method for Optimal Investment Under Relative Performance Criteria Among Heterogeneous Agents

Authors: Mathieu Laurière, Ludovic Tangpi, Xuchen Zhou

Abstract: Graphon games have been introduced to study games with many players who interact through a weighted graph of interaction. By passing to the limit, a game with a continuum of players is obtained, in which the interactions are through a graphon. In this paper, we focus on a graphon game for optimal investment under relative performance criteria, and we propose a deep learning method. The method buil… ▽ More Graphon games have been introduced to study games with many players who interact through a weighted graph of interaction. By passing to the limit, a game with a continuum of players is obtained, in which the interactions are through a graphon. In this paper, we focus on a graphon game for optimal investment under relative performance criteria, and we propose a deep learning method. The method builds upon two key ingredients: first, a characterization of Nash equilibria by forward-backward stochastic differential equations and, second, recent advances of machine learning algorithms for stochastic differential games. We provide numerical experiments on two different financial models. In each model, we compare the effect of several graphons, which correspond to different structures of interactions. △ Less

Submitted 30 March, 2024; v1 submitted 11 February, 2024; originally announced February 2024.

Learning Discrete-Time Major-Minor Mean Field Games

Authors: Kai Cui, Gökçe Dayanıklı, Mathieu Laurière, Matthieu Geist, Olivier Pietquin, Heinz Koeppl

Abstract: Recent techniques based on Mean Field Games (MFGs) allow the scalable analysis of multi-player games with many similar, rational agents. However, standard MFGs remain limited to homogeneous players that weakly influence each other, and cannot model major players that strongly influence other players, severely limiting the class of problems that can be handled. We propose a novel discrete time vers… ▽ More Recent techniques based on Mean Field Games (MFGs) allow the scalable analysis of multi-player games with many similar, rational agents. However, standard MFGs remain limited to homogeneous players that weakly influence each other, and cannot model major players that strongly influence other players, severely limiting the class of problems that can be handled. We propose a novel discrete time version of major-minor MFGs (M3FGs), along with a learning algorithm based on fictitious play and partitioning the probability simplex. Importantly, M3FGs generalize MFGs with common noise and can handle not only random exogeneous environment states but also major players. A key challenge is that the mean field is stochastic and not deterministic as in standard MFGs. Our theoretical investigation verifies both the M3FG model and its algorithmic solution, showing firstly the well-posedness of the M3FG model starting from a finite game of interest, and secondly convergence and approximation guarantees of the fictitious play algorithm. Then, we empirically verify the obtained theoretical results, ablating some of the theoretical assumptions made, and show successful equilibrium learning in three example problems. Overall, we establish a learning framework for a novel and broad class of tractable games. △ Less

Submitted 17 December, 2023; originally announced December 2023.

Comments: Accepted to AAAI 2024

From Nash Equilibrium to Social Optimum and vice versa: a Mean Field Perspective

Authors: Rene Carmona, Gokce Dayanikli, Francois Delarue, Mathieu Lauriere

Abstract: Mean field games (MFG) and mean field control (MFC) problems have been introduced to study large populations of strategic players. They correspond respectively to non-cooperative or cooperative scenarios, where the aim is to find the Nash equilibrium and social optimum. These frameworks provide approximate solutions to situations with a finite number of players and have found a wide range of appli… ▽ More Mean field games (MFG) and mean field control (MFC) problems have been introduced to study large populations of strategic players. They correspond respectively to non-cooperative or cooperative scenarios, where the aim is to find the Nash equilibrium and social optimum. These frameworks provide approximate solutions to situations with a finite number of players and have found a wide range of applications, from economics to biology and machine learning. In this paper, we study how the players can pass from a non-cooperative to a cooperative regime, and vice versa. The first direction is reminiscent of mechanism design, in which the game's definition is modified so that non-cooperative players reach an outcome similar to a cooperative scenario. The second direction studies how players that are initially cooperative gradually deviate from a social optimum to reach a Nash equilibrium when they decide to optimize their individual cost similar to the free rider phenomenon. To formalize these connections, we introduce two new classes of games which lie between MFG and MFC: $λ$-interpolated mean field games, in which the cost of an individual player is a $λ$-interpolation of the MFG and the MFC costs, and $p$-partial mean field games, in which a proportion $p$ of the population deviates from the social optimum by playing the game non-cooperatively. We conclude the paper by providing an algorithm for myopic players to learn a $p$-partial mean field equilibrium, and we illustrate it on a stylized model. △ Less

Submitted 16 December, 2023; originally announced December 2023.

Comments: 51 pages, 4 figures

MSC Class: 49N80; 49N90; 91A07; 91A10; 91A12; 91A15; 91A16

Convergence of Multi-Scale Reinforcement Q-Learning Algorithms for Mean Field Game and Control Problems

Authors: Andrea Angiuli, Jean-Pierre Fouque, Mathieu Laurière, Mengrui Zhang

Abstract: We establish the convergence of the unified two-timescale Reinforcement Learning (RL) algorithm presented in a previous work by Angiuli et al. This algorithm provides solutions to Mean Field Game (MFG) or Mean Field Control (MFC) problems depending on the ratio of two learning rates, one for the value function and the other for the mean field term. Our proof of convergence highlights the fact that… ▽ More We establish the convergence of the unified two-timescale Reinforcement Learning (RL) algorithm presented in a previous work by Angiuli et al. This algorithm provides solutions to Mean Field Game (MFG) or Mean Field Control (MFC) problems depending on the ratio of two learning rates, one for the value function and the other for the mean field term. Our proof of convergence highlights the fact that in the case of MFC several mean field distributions need to be updated and for this reason we present two separate algorithms, one for MFG and one for MFC. We focus on a setting with finite state and action spaces, discrete time and infinite horizon. The proofs of convergence rely on a generalization of the two-timescale approach of Borkar. The accuracy of approximation to the true solutions depends on the smoothing of the policies. We provide a numerical example illustrating the convergence. △ Less

Submitted 1 May, 2024; v1 submitted 11 December, 2023; originally announced December 2023.

Non-standard Stochastic Control with Nonlinear Feynman-Kac Costs

Authors: Rene Carmona, Mathieu Lauriere, Pierre-Louis Lions

Abstract: We consider the conditional control problem introduced by P.L. Lions in his lectures at the Collège de France in November 2016. In his lectures, Lions emphasized some of the major differences with the analysis of classical stochastic optimal control problems, and in so doing, raised the question of the possible differences between the value functions resulting from optimization over the class of M… ▽ More We consider the conditional control problem introduced by P.L. Lions in his lectures at the Collège de France in November 2016. In his lectures, Lions emphasized some of the major differences with the analysis of classical stochastic optimal control problems, and in so doing, raised the question of the possible differences between the value functions resulting from optimization over the class of Markovian controls as opposed to the general family of open loop controls. The goal of the paper is to elucidate this quandary and provide elements of response to Lions' original conjecture. First, we justify the mathematical formulation of the conditional control problem by the description of practical model from evolutionary biology. Next, we relax the original formulation by the introduction of \emph{soft} as opposed to hard killing, and using a \emph{mimicking} argument, we reduce the open loop optimization problem to an optimization over a specific class of feedback controls. After proving existence of optimal feedback control functions, we prove a superposition principle allowing us to recast the original stochastic control problems as deterministic control problems for dynamical systems of probability Gibbs measures. Next, we characterize the solutions by forward-backward systems of coupled non-linear Partial Differential Equations (PDEs) very much in the spirit of the Mean Field Game (MFG) systems. From there, we identify a common optimizer, proving the conjecture of equality of the value functions. Finally we illustrate the results by convincing numerical experiments. △ Less

Submitted 1 December, 2023; originally announced December 2023.

Comments: 64 pages, 2 figures

MSC Class: 60Hxx

Multi-population Mean Field Games with Multiple Major Players: Application to Carbon Emission Regulations

Authors: Gokce Dayanikli, Mathieu Lauriere

Abstract: In this paper, we propose and study a mean field game model with multiple populations of minor players and multiple major players, motivated by applications to the regulation of carbon emissions. Each population of minor players represent a large group of electricity producers and each major player represents a regulator. We first characterize the minor players equilibrium controls using forward-b… ▽ More In this paper, we propose and study a mean field game model with multiple populations of minor players and multiple major players, motivated by applications to the regulation of carbon emissions. Each population of minor players represent a large group of electricity producers and each major player represents a regulator. We first characterize the minor players equilibrium controls using forward-backward differential equations, and show existence and uniqueness of the minor players equilibrium. We then express the major players' equilibrium controls through analytical formulas given the other players' controls. Finally, we then provide a method to solve the Nash equilibrium between all the players, and we illustrate numerically the sensitivity of the model to its parameters. △ Less

Submitted 28 September, 2023; originally announced September 2023.

Deep Learning for Population-Dependent Controls in Mean Field Control Problems with Common Noise

Authors: Gokce Dayanikli, Mathieu Lauriere, Jiacheng Zhang

Abstract: In this paper, we propose several approaches to learn the optimal population-dependent controls in order to solve mean field control problems (MFC). Such policies enable us to solve MFC problems with forms of common noises at a level of generality that was not covered by existing methods. We analyze rigorously the theoretical convergence of the proposed approximation algorithms. Of particular inte… ▽ More In this paper, we propose several approaches to learn the optimal population-dependent controls in order to solve mean field control problems (MFC). Such policies enable us to solve MFC problems with forms of common noises at a level of generality that was not covered by existing methods. We analyze rigorously the theoretical convergence of the proposed approximation algorithms. Of particular interest for its simplicity of implementation is the $N$-particle approximation. The effectiveness and the flexibility of our algorithms is supported by numerical experiments comparing several combinations of distribution approximation techniques and neural network architectures. We use three different benchmark problems from the literature: a systemic risk model, a price impact model, and a crowd motion model. We first show that our proposed algorithms converge to the correct solution in an explicitly solvable MFC problem. Then, we show that population-dependent controls outperform state-dependent controls. Along the way, we show that specific neural network architectures can improve the learning further. △ Less

Submitted 20 November, 2023; v1 submitted 7 June, 2023; originally announced June 2023.

Comments: 23 pages, 6 figures

Machine Learning architectures for price formation models with common noise

Authors: Diogo Gomes, Julian Gutierrez, Mathieu Laurière

Abstract: We propose a machine learning method to solve a mean-field game price formation model with common noise. This involves determining the price of a commodity traded among rational agents subject to a market clearing condition imposed by random supply, which presents additional challenges compared to the deterministic counterpart. Our approach uses a dual recurrent neural network architecture encodin… ▽ More We propose a machine learning method to solve a mean-field game price formation model with common noise. This involves determining the price of a commodity traded among rational agents subject to a market clearing condition imposed by random supply, which presents additional challenges compared to the deterministic counterpart. Our approach uses a dual recurrent neural network architecture encoding noise dependence and a particle approximation of the mean-field model with a single loss function optimized by adversarial training. We provide a posteriori estimates for convergence and illustrate our method through numerical experiments. △ Less

Submitted 27 May, 2023; originally announced May 2023.

Comments: 6 pages, 3 figures, conference paper

MSC Class: 91A15; 91-08; 68T07; 60H10; 65C20

Recent Developments in Machine Learning Methods for Stochastic Control and Games

Authors: Ruimeng Hu, Mathieu Laurière

Abstract: Stochastic optimal control and games have a wide range of applications, from finance and economics to social sciences, robotics, and energy management. Many real-world applications involve complex models that have driven the development of sophisticated numerical methods. Recently, computational methods based on machine learning have been developed for solving stochastic control problems and games… ▽ More Stochastic optimal control and games have a wide range of applications, from finance and economics to social sciences, robotics, and energy management. Many real-world applications involve complex models that have driven the development of sophisticated numerical methods. Recently, computational methods based on machine learning have been developed for solving stochastic control problems and games. In this review, we focus on deep learning methods that have unlocked the possibility of solving such problems, even in high dimensions or when the structure is very complex, beyond what traditional numerical methods can achieve. We consider mostly the continuous time and continuous space setting. Many of the new approaches build on recent neural-network-based methods for solving high-dimensional partial differential equations or backward stochastic differential equations, or on model-free reinforcement learning for Markov decision processes that have led to breakthrough results. This paper provides an introduction to these methods and summarizes the state-of-the-art works at the crossroad of machine learning and stochastic control and games. △ Less

Submitted 11 March, 2024; v1 submitted 17 March, 2023; originally announced March 2023.

Actor-Critic learning for mean-field control in continuous time

Authors: Noufel Frikha, Maximilien Germain, Mathieu Laurière, Huyên Pham, Xuanye Song

Abstract: We study policy gradient for mean-field control in continuous time in a reinforcement learning setting. By considering randomised policies with entropy regularisation, we derive a gradient expectation representation of the value function, which is amenable to actor-critic type algorithms, where the value functions and the policies are learnt alternately based on observation samples of the state… ▽ More We study policy gradient for mean-field control in continuous time in a reinforcement learning setting. By considering randomised policies with entropy regularisation, we derive a gradient expectation representation of the value function, which is amenable to actor-critic type algorithms, where the value functions and the policies are learnt alternately based on observation samples of the state and model-free estimation of the population state distribution, either by offline or online learning. In the linear-quadratic mean-field framework, we obtain an exact parametrisation of the actor and critic functions defined on the Wasserstein space. Finally, we illustrate the results of our algorithms with some numerical experiments on concrete examples. △ Less

Submitted 13 March, 2023; originally announced March 2023.

Deep Learning for Mean Field Optimal Transport

Authors: Sebastian Baudelet, Brieuc Frénais, Mathieu Laurière, Amal Machtalay, Yuchen Zhu

Abstract: Mean field control (MFC) problems have been introduced to study social optima in very large populations of strategic agents. The main idea is to consider an infinite population and to simplify the analysis by using a mean field approximation. These problems can also be viewed as optimal control problems for McKean-Vlasov dynamics. They have found applications in a wide range of fields, from econom… ▽ More Mean field control (MFC) problems have been introduced to study social optima in very large populations of strategic agents. The main idea is to consider an infinite population and to simplify the analysis by using a mean field approximation. These problems can also be viewed as optimal control problems for McKean-Vlasov dynamics. They have found applications in a wide range of fields, from economics and finance to social sciences and engineering. Usually, the goal for the agents is to minimize a total cost which consists in the integral of a running cost plus a terminal cost. In this work, we consider MFC problems in which there is no terminal cost but, instead, the terminal distribution is prescribed. We call such problems mean field optimal transport problems since they can be viewed as a generalization of classical optimal transport problems when mean field interactions occur in the dynamics or the running cost function. We propose three numerical methods based on neural networks. The first one is based on directly learning an optimal control. The second one amounts to solve a forward-backward PDE system characterizing the solution. The third one relies on a primal-dual approach. We illustrate these methods with numerical experiments conducted on two families of examples. △ Less

Submitted 28 February, 2023; originally announced February 2023.

A Machine Learning Method for Stackelberg Mean Field Games

Authors: Gokce Dayanikli, Mathieu Lauriere

Abstract: We propose a single-level numerical approach to solve Stackelberg mean field game (MFG) problems. In Stackelberg MFG, an infinite population of agents play a non-cooperative game and choose their controls to optimize their individual objectives while interacting with the principal and other agents through the population distribution. The principal can influence the mean field Nash equilibrium at t… ▽ More We propose a single-level numerical approach to solve Stackelberg mean field game (MFG) problems. In Stackelberg MFG, an infinite population of agents play a non-cooperative game and choose their controls to optimize their individual objectives while interacting with the principal and other agents through the population distribution. The principal can influence the mean field Nash equilibrium at the population level through policies, and she optimizes her own objective, which depends on the population distribution. This leads to a bi-level problem between the principal and mean field of agents that cannot be solved using traditional methods for MFGs. We propose a reformulation of this problem as a single-level mean field optimal control problem through a penalization approach. We prove convergence of the reformulated problem to the original problem. We propose a machine learning method based on (feed-forward and recurrent) neural networks and illustrate it on several examples from the literature. △ Less

Submitted 22 April, 2024; v1 submitted 20 February, 2023; originally announced February 2023.

Comments: 47 pages, 9 figures, 4 tables

MSC Class: 49N90; 91A13; 91A15; 62M45

Reinforcement Learning for Intra-and-Inter-Bank Borrowing and Lending Mean Field Control Game

Authors: Andrea Angiuli, Nils Detering, Jean-Pierre Fouque, Mathieu Laurière, Jimin Lin

Abstract: We propose a mean field control game model for the intra-and-inter-bank borrowing and lending problem. This framework allows to study the competitive game arising between groups of collaborative banks. The solution is provided in terms of an asymptotic Nash equilibrium between the groups in the infinite horizon. A three-timescale reinforcement learning algorithm is applied to learn the optimal bor… ▽ More We propose a mean field control game model for the intra-and-inter-bank borrowing and lending problem. This framework allows to study the competitive game arising between groups of collaborative banks. The solution is provided in terms of an asymptotic Nash equilibrium between the groups in the infinite horizon. A three-timescale reinforcement learning algorithm is applied to learn the optimal borrowing and lending strategy in a data driven way when the model is unknown. An empirical numerical analysis shows the importance of the three-timescale, the impact of the exploration strategy when the model is unknown, and the convergence of the algorithm. △ Less

Submitted 7 July, 2022; originally announced July 2022.

Learning in Mean Field Games: A Survey

Authors: Mathieu Laurière, Sarah Perrin, Julien Pérolat, Sertan Girgin, Paul Muller, Romuald Élie, Matthieu Geist, Olivier Pietquin

Abstract: Non-cooperative and cooperative games with a very large number of players have many applications but remain generally intractable when the number of players increases. Introduced by Lasry and Lions, and Huang, Caines and Malhamé, Mean Field Games (MFGs) rely on a mean-field approximation to allow the number of players to grow to infinity. Traditional methods for solving these games generally rely… ▽ More Non-cooperative and cooperative games with a very large number of players have many applications but remain generally intractable when the number of players increases. Introduced by Lasry and Lions, and Huang, Caines and Malhamé, Mean Field Games (MFGs) rely on a mean-field approximation to allow the number of players to grow to infinity. Traditional methods for solving these games generally rely on solving partial or stochastic differential equations with a full knowledge of the model. Recently, Reinforcement Learning (RL) has appeared promising to solve complex problems at scale. The combination of RL and MFGs is promising to solve games at a very large scale both in terms of population size and environment complexity. In this survey, we review the quickly growing recent literature on RL methods to learn equilibria and social optima in MFGs. We first identify the most common settings (static, stationary, and evolutive) of MFGs. We then present a general framework for classical iterative methods (based on best-response computation or policy evaluation) to solve MFGs in an exact way. Building on these algorithms and the connection with Markov Decision Processes, we explain how RL can be used to learn MFG solutions in a model-free way. Last, we present numerical illustrations on a benchmark problem, and conclude with some perspectives. △ Less

Submitted 26 July, 2024; v1 submitted 25 May, 2022; originally announced May 2022.

Reinforcement Learning Algorithm for Mixed Mean Field Control Games

Authors: Andrea Angiuli, Nils Detering, Jean-Pierre Fouque, Mathieu Lauriere, Jimin Lin

Abstract: We present a new combined \textit{mean field control game} (MFCG) problem which can be interpreted as a competitive game between collaborating groups and its solution as a Nash equilibrium between groups. Players coordinate their strategies within each group. An example is a modification of the classical trader's problem. Groups of traders maximize their wealth. They face cost for their transactio… ▽ More We present a new combined \textit{mean field control game} (MFCG) problem which can be interpreted as a competitive game between collaborating groups and its solution as a Nash equilibrium between groups. Players coordinate their strategies within each group. An example is a modification of the classical trader's problem. Groups of traders maximize their wealth. They face cost for their transactions, for their own terminal positions, and for the average holding within their group. The asset price is impacted by the trades of all agents. We propose a three-timescale reinforcement learning algorithm to approximate the solution of such MFCG problems. We test the algorithm on benchmark linear-quadratic specifications for which we provide analytic solutions. △ Less

Submitted 15 February, 2023; v1 submitted 4 May, 2022; originally announced May 2022.

MSC Class: 91A16; 68Q32

Machine Learning architectures for price formation models

Authors: Diogo Gomes, Julián Gutiérrez, Mathieu Laurière

Abstract: Here, we study machine learning (ML) architectures to solve a mean-field games (MFGs) system arising in price formation models. We formulate a training process that relies on a min-max characterization of the optimal control and price variables. Our main theoretical contribution is the development of a posteriori estimates as a tool to evaluate the convergence of the training process. We illustrat… ▽ More Here, we study machine learning (ML) architectures to solve a mean-field games (MFGs) system arising in price formation models. We formulate a training process that relies on a min-max characterization of the optimal control and price variables. Our main theoretical contribution is the development of a posteriori estimates as a tool to evaluate the convergence of the training process. We illustrate our results with numerical experiments for linear dynamics and both quadratic and non-quadratic models. △ Less

Submitted 25 January, 2023; v1 submitted 8 April, 2022; originally announced April 2022.

MSC Class: 35Q89; 49N80; 68T07

Scalable Deep Reinforcement Learning Algorithms for Mean Field Games

Authors: Mathieu Laurière, Sarah Perrin, Sertan Girgin, Paul Muller, Ayush Jain, Theophile Cabannes, Georgios Piliouras, Julien Pérolat, Romuald Élie, Olivier Pietquin, Matthieu Geist

Abstract: Mean Field Games (MFGs) have been introduced to efficiently approximate games with very large populations of strategic agents. Recently, the question of learning equilibria in MFGs has gained momentum, particularly using model-free reinforcement learning (RL) methods. One limiting factor to further scale up using RL is that existing algorithms to solve MFGs require the mixing of approximated quant… ▽ More Mean Field Games (MFGs) have been introduced to efficiently approximate games with very large populations of strategic agents. Recently, the question of learning equilibria in MFGs has gained momentum, particularly using model-free reinforcement learning (RL) methods. One limiting factor to further scale up using RL is that existing algorithms to solve MFGs require the mixing of approximated quantities such as strategies or $q$-values. This is far from being trivial in the case of non-linear function approximation that enjoy good generalization properties, e.g. neural networks. We propose two methods to address this shortcoming. The first one learns a mixed strategy from distillation of historical data into a neural network and is applied to the Fictitious Play algorithm. The second one is an online mixing method based on regularization that does not require memorizing historical data or previous estimates. It is used to extend Online Mirror Descent. We demonstrate numerically that these methods efficiently enable the use of Deep RL algorithms to solve various MFGs. In addition, we show that these methods outperform SotA baselines from the literature. △ Less

Submitted 17 June, 2022; v1 submitted 22 March, 2022; originally announced March 2022.

Solving N-player dynamic routing games with congestion: a mean field approach

Authors: Theophile Cabannes, Mathieu Lauriere, Julien Perolat, Raphael Marinier, Sertan Girgin, Sarah Perrin, Olivier Pietquin, Alexandre M. Bayen, Eric Goubault, Romuald Elie

Abstract: The recent emergence of navigational tools has changed traffic patterns and has now enabled new types of congestion-aware routing control like dynamic road pricing. Using the fundamental diagram of traffic flows - applied in macroscopic and mesoscopic traffic modeling - the article introduces a new N-player dynamic routing game with explicit congestion dynamics. The model is well-posed and can rep… ▽ More The recent emergence of navigational tools has changed traffic patterns and has now enabled new types of congestion-aware routing control like dynamic road pricing. Using the fundamental diagram of traffic flows - applied in macroscopic and mesoscopic traffic modeling - the article introduces a new N-player dynamic routing game with explicit congestion dynamics. The model is well-posed and can reproduce heterogeneous departure times and congestion spill back phenomena. However, as Nash equilibrium computations are PPAD-complete, solving the game becomes intractable for large but realistic numbers of vehicles N. Therefore, the corresponding mean field game is also introduced. Experiments were performed on several classical benchmark networks of the traffic community: the Pigou, Braess, and Sioux Falls networks with heterogeneous origin, destination and departure time tuples. The Pigou and the Braess examples reveal that the mean field approximation is generally very accurate and computationally efficient as soon as the number of vehicles exceeds a few dozen. On the Sioux Falls network (76 links, 100 time steps), this approach enables learning traffic dynamics with more than 14,000 vehicles. △ Less

Submitted 27 October, 2021; v1 submitted 22 October, 2021; originally announced October 2021.

Policy iteration method for time-dependent Mean Field Games systems with non-separable Hamiltonians

Authors: Mathieu Laurière, Jiahao Song, Qing Tang

Abstract: We introduce two algorithms based on a policy iteration method to numerically solve time-dependent Mean Field Game systems of partial differential equations with non-separable Hamiltonians. We prove the convergence of such algorithms in sufficiently small time intervals with Banach fixed point method. Moreover, we prove that the convergence rates are linear. We illustrate our theoretical results b… ▽ More We introduce two algorithms based on a policy iteration method to numerically solve time-dependent Mean Field Game systems of partial differential equations with non-separable Hamiltonians. We prove the convergence of such algorithms in sufficiently small time intervals with Banach fixed point method. Moreover, we prove that the convergence rates are linear. We illustrate our theoretical results by numerical examples, and we discuss the performance of the proposed algorithms. △ Less

Submitted 30 September, 2022; v1 submitted 6 October, 2021; originally announced October 2021.

Comments: arXiv admin note: text overlap with arXiv:2007.04818 by other authors

MSC Class: 49N70; 35Q91; 91A13; 49D10

Generalization in Mean Field Games by Learning Master Policies

Authors: Sarah Perrin, Mathieu Laurière, Julien Pérolat, Romuald Élie, Matthieu Geist, Olivier Pietquin

Abstract: Mean Field Games (MFGs) can potentially scale multi-agent systems to extremely large populations of agents. Yet, most of the literature assumes a single initial distribution for the agents, which limits the practical applications of MFGs. Machine Learning has the potential to solve a wider diversity of MFG problems thanks to generalizations capacities. We study how to leverage these generalization… ▽ More Mean Field Games (MFGs) can potentially scale multi-agent systems to extremely large populations of agents. Yet, most of the literature assumes a single initial distribution for the agents, which limits the practical applications of MFGs. Machine Learning has the potential to solve a wider diversity of MFG problems thanks to generalizations capacities. We study how to leverage these generalization properties to learn policies enabling a typical agent to behave optimally against any population distribution. In reference to the Master equation in MFGs, we coin the term ``Master policies'' to describe them and we prove that a single Master policy provides a Nash equilibrium, whatever the initial distribution. We propose a method to learn such Master policies. Our approach relies on three ingredients: adding the current population distribution as part of the observation, approximating Master policies with neural networks, and training via Reinforcement Learning and Fictitious Play. We illustrate on numerical examples not only the efficiency of the learned Master policy but also its generalization capabilities beyond the distributions used for training. △ Less

Submitted 20 September, 2021; originally announced September 2021.

Performance of a Markovian neural network versus dynamic programming on a fishing control problem

Authors: Mathieu Laurière, Gilles Pagès, Olivier Pironneau

Abstract: Fishing quotas are unpleasant but efficient to control the productivity of a fishing site. A popular model has a stochastic differential equation for the biomass on which a stochastic dynamic programming or a Hamilton-Jacobi-Bellman algorithm can be used to find the stochastic control -- the fishing quota. We compare the solutions obtained by dynamic programming against those obtained with a neura… ▽ More Fishing quotas are unpleasant but efficient to control the productivity of a fishing site. A popular model has a stochastic differential equation for the biomass on which a stochastic dynamic programming or a Hamilton-Jacobi-Bellman algorithm can be used to find the stochastic control -- the fishing quota. We compare the solutions obtained by dynamic programming against those obtained with a neural network which preserves the Markov property of the solution. The method is extended to a similar multi species model to check its robustness in high dimension. △ Less

Submitted 14 September, 2021; originally announced September 2021.

Deep Learning for Mean Field Games and Mean Field Control with Applications to Finance

Authors: René Carmona, Mathieu Laurière

Abstract: Financial markets and more generally macro-economic models involve a large number of individuals interacting through variables such as prices resulting from the aggregate behavior of all the agents. Mean field games have been introduced to study Nash equilibria for such problems in the limit when the number of players is infinite. The theory has been extensively developed in the past decade, using… ▽ More Financial markets and more generally macro-economic models involve a large number of individuals interacting through variables such as prices resulting from the aggregate behavior of all the agents. Mean field games have been introduced to study Nash equilibria for such problems in the limit when the number of players is infinite. The theory has been extensively developed in the past decade, using both analytical and probabilistic tools, and a wide range of applications have been discovered, from economics to crowd motion. More recently the interaction with machine learning has attracted a growing interest. This aspect is particularly relevant to solve very large games with complex structures, in high dimension or with common sources of randomness. In this chapter, we review the literature on the interplay between mean field games and deep learning, with a focus on three families of methods. A special emphasis is given to financial applications. △ Less

Submitted 9 July, 2021; originally announced July 2021.

Reinforcement Learning for Mean Field Games, with Applications to Economics

Authors: Andrea Angiuli, Jean-Pierre Fouque, Mathieu Lauriere

Abstract: Mean field games (MFG) and mean field control problems (MFC) are frameworks to study Nash equilibria or social optima in games with a continuum of agents. These problems can be used to approximate competitive or cooperative games with a large finite number of agents and have found a broad range of applications, in particular in economics. In recent years, the question of learning in MFG and MFC ha… ▽ More Mean field games (MFG) and mean field control problems (MFC) are frameworks to study Nash equilibria or social optima in games with a continuum of agents. These problems can be used to approximate competitive or cooperative games with a large finite number of agents and have found a broad range of applications, in particular in economics. In recent years, the question of learning in MFG and MFC has garnered interest, both as a way to compute solutions and as a way to model how large populations of learners converge to an equilibrium. Of particular interest is the setting where the agents do not know the model, which leads to the development of reinforcement learning (RL) methods. After reviewing the literature on this topic, we present a two timescale approach with RL for MFG and MFC, which relies on a unified Q-learning algorithm. The main novelty of this method is to simultaneously update an action-value function and a distribution but with different rates, in a model-free fashion. Depending on the ratio of the two learning rates, the algorithm learns either the MFG or the MFC solution. To illustrate this method, we apply it to a mean field problem of accumulated consumption in finite horizon with HARA utility function, and to a trader's optimal liquidation problem. △ Less

Submitted 25 June, 2021; originally announced June 2021.

Finite State Graphon Games with Applications to Epidemics

Authors: Alexander Aurell, Rene Carmona, Gokce Dayanikli, Mathieu Lauriere

Abstract: We consider a game for a continuum of non-identical players evolving on a finite state space. Their heterogeneous interactions are represented by a graphon, which can be viewed as the limit of a dense random graph. The player's transition rates between the states depend on their own control and the interaction strengths with the other players. We develop a rigorous mathematical framework for this… ▽ More We consider a game for a continuum of non-identical players evolving on a finite state space. Their heterogeneous interactions are represented by a graphon, which can be viewed as the limit of a dense random graph. The player's transition rates between the states depend on their own control and the interaction strengths with the other players. We develop a rigorous mathematical framework for this game and analyze Nash equilibria. We provide a sufficient condition for a Nash equilibrium and prove existence of solutions to a continuum of fully coupled forward-backward ordinary differential equations characterizing equilibria. Moreover, we propose a numerical approach based on machine learning tools and show experimental results on different applications to compartmental models in epidemiology. △ Less

Submitted 14 June, 2021; originally announced June 2021.

Comments: 32 pages, 5 figures, 5 tables

Numerical Methods for Mean Field Games and Mean Field Type Control

Authors: Mathieu Lauriere

Abstract: Mean Field Games (MFG) have been introduced to tackle games with a large number of competing players. Considering the limit when the number of players is infinite, Nash equilibria are studied by considering the interaction of a typical player with the population's distribution. The situation in which the players cooperate corresponds to Mean Field Control (MFC) problems, which can also be viewed a… ▽ More Mean Field Games (MFG) have been introduced to tackle games with a large number of competing players. Considering the limit when the number of players is infinite, Nash equilibria are studied by considering the interaction of a typical player with the population's distribution. The situation in which the players cooperate corresponds to Mean Field Control (MFC) problems, which can also be viewed as optimal control problems driven by a McKean-Vlasov dynamics. These two types of problems have found a wide range of potential applications, for which numerical methods play a key role since most models do not have analytical solutions. In these notes, we review several aspects of numerical methods for MFG and MFC. We start by presenting some heuristics in a basic linear-quadratic setting. We then discuss numerical schemes for forward-backward systems of partial differential equations (PDEs), optimization techniques for variational problems driven by a Kolmogorov-Fokker-Planck PDE, an approach based on a monotone operator viewpoint, and stochastic methods relying on machine learning tools. △ Less

Submitted 11 June, 2021; originally announced June 2021.

Stochastic Graphon Games: II. The Linear-Quadratic Case

Authors: Alexander Aurell, Rene Carmona, Mathieu Lauriere

Abstract: In this paper, we analyze linear-quadratic stochastic differential games with a continuum of players interacting through graphon aggregates, each state being subject to idiosyncratic Brownian shocks. The major technical issue is the joint measurability of the player state trajectories with respect to samples and player labels, which is required to compute for example costs involving the graphon ag… ▽ More In this paper, we analyze linear-quadratic stochastic differential games with a continuum of players interacting through graphon aggregates, each state being subject to idiosyncratic Brownian shocks. The major technical issue is the joint measurability of the player state trajectories with respect to samples and player labels, which is required to compute for example costs involving the graphon aggregate. To resolve this issue we set the game in a Fubini extension of a product probability space. We provide conditions under which the graphon aggregates are deterministic and the linear state equation is uniquely solvable for all players in the continuum. The Pontryagin maximum principle yields equilibrium conditions for the graphon game in the form of a forward-backward stochastic differential equation, for which we establish existence and uniqueness. We then study how graphon games approximate games with finitely many players over graphs with random weights. We illustrate some of the results with a numerical example. △ Less

Submitted 26 May, 2021; originally announced May 2021.

Comments: 33 pages, 1 figure

MSC Class: 91A15; 91A0; 60H10; 60H20; 60G15; 28E05

DeepSets and their derivative networks for solving symmetric PDEs

Authors: Maximilien Germain, Mathieu Laurière, Huyên Pham, Xavier Warin

Abstract: Machine learning methods for solving nonlinear partial differential equations (PDEs) are hot topical issues, and different algorithms proposed in the literature show efficient numerical approximation in high dimension. In this paper, we introduce a class of PDEs that are invariant to permutations, and called symmetric PDEs. Such problems are widespread, ranging from cosmology to quantum mechanics,… ▽ More Machine learning methods for solving nonlinear partial differential equations (PDEs) are hot topical issues, and different algorithms proposed in the literature show efficient numerical approximation in high dimension. In this paper, we introduce a class of PDEs that are invariant to permutations, and called symmetric PDEs. Such problems are widespread, ranging from cosmology to quantum mechanics, and option pricing/hedging in multi-asset market with exchangeable payoff. Our main application comes actually from the particles approximation of mean-field control problems. We design deep learning algorithms based on certain types of neural networks, named PointNet and DeepSet (and their associated derivative networks), for computing simultaneously an approximation of the solution and its gradient to symmetric PDEs. We illustrate the performance and accuracy of the PointNet/DeepSet networks compared to classical feedforward ones, and provide several numerical results of our algorithm for the examples of a mean-field systemic risk, mean-variance problem and a min/max linear quadratic McKean-Vlasov control problem. △ Less

Submitted 4 January, 2022; v1 submitted 1 March, 2021; originally announced March 2021.

Journal ref: Journal of Scientific Computing, Springer Verlag, In press

Mean Field Models to Regulate Carbon Emissions in Electricity Production

Authors: Rene Carmona, Gokce Dayanikli, Mathieu Lauriere

Abstract: The most serious threat to ecosystems is the global climate change fueled by the uncontrolled increase in carbon emissions. In this project, we use mean field control and mean field game models to analyze and inform the decisions of electricity producers on how much renewable sources of production ought to be used in the presence of a carbon tax. The trade-off between higher revenues from producti… ▽ More The most serious threat to ecosystems is the global climate change fueled by the uncontrolled increase in carbon emissions. In this project, we use mean field control and mean field game models to analyze and inform the decisions of electricity producers on how much renewable sources of production ought to be used in the presence of a carbon tax. The trade-off between higher revenues from production and the negative externality of carbon emissions is quantified for each producer who needs to balance in real time reliance on reliable but polluting (fossil fuel) thermal power stations versus investing in and depending upon clean production from uncertain wind and solar technologies. We compare the impacts of these decisions in two different scenarios: 1) the producers are competitive and hopefully reach a Nash Equilibrium; 2) they cooperate and reach a Social Optimum. We first prove that both problems have a unique solution using forward-backward systems of stochastic differential equations. We then illustrate with numerical experiments the producers' behavior in each scenario. We further introduce and analyze the impact of a regulator in control of the carbon tax policy, and we study the resulting Stackelberg equilibrium with the field of producers. △ Less

Submitted 3 July, 2021; v1 submitted 18 February, 2021; originally announced February 2021.

Comments: 29 pages, 6 figures, 6 algorithms, 2 tables

MSC Class: 49N80; 91A16; 91B76; 49N90; 91A07

Optimal incentives to mitigate epidemics: a Stackelberg mean field game approach

Authors: Alexander Aurell, Rene Carmona, Gokce Dayanikli, Mathieu Lauriere

Abstract: Motivated by models of epidemic control in large populations, we consider a Stackelberg mean field game model between a principal and a mean field of agents evolving on a finite state space. The agents play a non-cooperative game in which they can control their transition rates between states to minimize an individual cost. The principal can influence the resulting Nash equilibrium through incenti… ▽ More Motivated by models of epidemic control in large populations, we consider a Stackelberg mean field game model between a principal and a mean field of agents evolving on a finite state space. The agents play a non-cooperative game in which they can control their transition rates between states to minimize an individual cost. The principal can influence the resulting Nash equilibrium through incentives so as to optimize its own objective. We analyze this game using a probabilistic approach. We then propose an application to an epidemic model of SIR type in which the agents control their interaction rate and the principal is a regulator acting with non pharmaceutical interventions. To compute the solutions, we propose an innovative numerical approach based on Monte Carlo simulations and machine learning tools for stochastic optimization. We conclude with numerical experiments by illustrating the impact of the agents' and the regulator's optimal decisions in two models: a basic SIR model with semi-explicit solutions and a more complex model with a larger state space. △ Less

Submitted 24 May, 2021; v1 submitted 5 November, 2020; originally announced November 2020.

Comments: 31 pages, 10 figures, 3 algorithms, 3 tables

MSC Class: 92D30; 49N90; 91A13; 91A15; 62M45

Policy Optimization for Linear-Quadratic Zero-Sum Mean-Field Type Games

Authors: René Carmona, Kenza Hamidouche, Mathieu Laurière, Zongjun Tan

Abstract: In this paper, zero-sum mean-field type games (ZSMFTG) with linear dynamics and quadratic utility are studied under infinite-horizon discounted utility function. ZSMFTG are a class of games in which two decision makers whose utilities sum to zero, compete to influence a large population of agents. In particular, the case in which the transition and utility functions depend on the state, the action… ▽ More In this paper, zero-sum mean-field type games (ZSMFTG) with linear dynamics and quadratic utility are studied under infinite-horizon discounted utility function. ZSMFTG are a class of games in which two decision makers whose utilities sum to zero, compete to influence a large population of agents. In particular, the case in which the transition and utility functions depend on the state, the action of the controllers, and the mean of the state and the actions, is investigated. The game is analyzed and explicit expressions for the Nash equilibrium strategies are derived. Moreover, two policy optimization methods that rely on policy gradient are proposed for both model-based and sample-based frameworks. In the first case, the gradients are computed exactly using the model whereas they are estimated using Monte-Carlo simulations in the second case. Numerical experiments show the convergence of the two players' controls as well as the utility function when the two algorithms are used in different scenarios. △ Less

Submitted 2 September, 2020; originally announced September 2020.

Comments: arXiv admin note: text overlap with arXiv:2009.00578

Linear-Quadratic Zero-Sum Mean-Field Type Games: Optimality Conditions and Policy Optimization

Authors: René Carmona, Kenza Hamidouche, Mathieu Laurière, Zongjun Tan

Abstract: In this paper, zero-sum mean-field type games (ZSMFTG) with linear dynamics and quadratic cost are studied under infinite-horizon discounted utility function. ZSMFTG are a class of games in which two decision makers whose utilities sum to zero, compete to influence a large population of indistinguishable agents. In particular, the case in which the transition and utility functions depend on the st… ▽ More In this paper, zero-sum mean-field type games (ZSMFTG) with linear dynamics and quadratic cost are studied under infinite-horizon discounted utility function. ZSMFTG are a class of games in which two decision makers whose utilities sum to zero, compete to influence a large population of indistinguishable agents. In particular, the case in which the transition and utility functions depend on the state, the action of the controllers, and the mean of the state and the actions, is investigated. The optimality conditions of the game are analysed for both open-loop and closed-loop controls, and explicit expressions for the Nash equilibrium strategies are derived. Moreover, two policy optimization methods that rely on policy gradient are proposed for both model-based and sample-based frameworks. In the model-based case, the gradients are computed exactly using the model, whereas they are estimated using Monte-Carlo simulations in the sample-based case. Numerical experiments are conducted to show the convergence of the utility function as well as the two players' controls. △ Less

Submitted 1 September, 2020; originally announced September 2020.

Fictitious Play for Mean Field Games: Continuous Time Analysis and Applications

Authors: Sarah Perrin, Julien Perolat, Mathieu Laurière, Matthieu Geist, Romuald Elie, Olivier Pietquin

Abstract: In this paper, we deepen the analysis of continuous time Fictitious Play learning algorithm to the consideration of various finite state Mean Field Game settings (finite horizon, $γ$-discounted), allowing in particular for the introduction of an additional common noise. We first present a theoretical convergence analysis of the continuous time Fictitious Play process and prove that the induced e… ▽ More In this paper, we deepen the analysis of continuous time Fictitious Play learning algorithm to the consideration of various finite state Mean Field Game settings (finite horizon, $γ$-discounted), allowing in particular for the introduction of an additional common noise. We first present a theoretical convergence analysis of the continuous time Fictitious Play process and prove that the induced exploitability decreases at a rate $O(\frac{1}{t})$. Such analysis emphasizes the use of exploitability as a relevant metric for evaluating the convergence towards a Nash equilibrium in the context of Mean Field Games. These theoretical contributions are supported by numerical experiments provided in either model-based or model-free settings. We provide hereby for the first time converging learning dynamics for Mean Field Games in the presence of common noise. △ Less

Submitted 26 October, 2020; v1 submitted 5 July, 2020; originally announced July 2020.

Unified Reinforcement Q-Learning for Mean Field Game and Control Problems

Authors: Andrea Angiuli, Jean-Pierre Fouque, Mathieu Laurière

Abstract: We present a Reinforcement Learning (RL) algorithm to solve infinite horizon asymptotic Mean Field Game (MFG) and Mean Field Control (MFC) problems. Our approach can be described as a unified two-timescale Mean Field Q-learning: The \emph{same} algorithm can learn either the MFG or the MFC solution by simply tuning the ratio of two learning parameters. The algorithm is in discrete time and space w… ▽ More We present a Reinforcement Learning (RL) algorithm to solve infinite horizon asymptotic Mean Field Game (MFG) and Mean Field Control (MFC) problems. Our approach can be described as a unified two-timescale Mean Field Q-learning: The \emph{same} algorithm can learn either the MFG or the MFC solution by simply tuning the ratio of two learning parameters. The algorithm is in discrete time and space where the agent not only provides an action to the environment but also a distribution of the state in order to take into account the mean field feature of the problem. Importantly, we assume that the agent can not observe the population's distribution and needs to estimate it in a model-free manner. The asymptotic MFG and MFC problems are also presented in continuous time and space, and compared with classical (non-asymptotic or stationary) MFG and MFC problems. They lead to explicit solutions in the linear-quadratic (LQ) case that are used as benchmarks for the results of our algorithm. △ Less

Submitted 31 May, 2021; v1 submitted 24 June, 2020; originally announced June 2020.

Learning a functional control for high-frequency finance

Authors: Laura Leal, Mathieu Laurière, Charles-Albert Lehalle

Abstract: We use a deep neural network to generate controllers for optimal trading on high frequency data. For the first time, a neural network learns the mapping between the preferences of the trader, i.e. risk aversion parameters, and the optimal controls. An important challenge in learning this mapping is that in intraday trading, trader's actions influence price dynamics in closed loop via the market im… ▽ More We use a deep neural network to generate controllers for optimal trading on high frequency data. For the first time, a neural network learns the mapping between the preferences of the trader, i.e. risk aversion parameters, and the optimal controls. An important challenge in learning this mapping is that in intraday trading, trader's actions influence price dynamics in closed loop via the market impact. The exploration--exploitation tradeoff generated by the efficient execution is addressed by tuning the trader's preferences to ensure long enough trajectories are produced during the learning phase. The issue of scarcity of financial data is solved by transfer learning: the neural network is first trained on trajectories generated thanks to a Monte-Carlo scheme, leading to a good initialization before training on historical trajectories. Moreover, to answer to genuine requests of financial regulators on the explainability of machine learning generated controls, we project the obtained "blackbox controls" on the space usually spanned by the closed-form solution of the stylized optimal trading problem, leading to a transparent structure. For more realistic loss functions that have no closed-form solution, we show that the average distance between the generated controls and their explainable version remains small. This opens the door to the acceptance of ML-generated controls by financial regulators. △ Less

Submitted 11 February, 2021; v1 submitted 16 June, 2020; originally announced June 2020.

Comments: 24 pages, 21 figures

Convergence of large population games to mean field games with interaction through the controls

Authors: Mathieu Laurière, Ludovic Tangpi

Abstract: This work considers stochastic differential games with a large number of players, whose costs and dynamics interact through the empirical distribution of both their states and their controls. We develop a new framework to prove convergence of finite-player games to the asymptotic mean field game. Our approach is based on the concept of propagation of chaos for forward and backward weakly interacti… ▽ More This work considers stochastic differential games with a large number of players, whose costs and dynamics interact through the empirical distribution of both their states and their controls. We develop a new framework to prove convergence of finite-player games to the asymptotic mean field game. Our approach is based on the concept of propagation of chaos for forward and backward weakly interacting particles which we investigate by stochastic analysis methods, and which appear to be of independent interest. These propagation of chaos arguments allow to derive moment and concentration bounds for the convergence of Nash equilibria. △ Less

Submitted 22 March, 2022; v1 submitted 17 April, 2020; originally announced April 2020.

Mean Field Games and Applications: Numerical Aspects

Authors: Yves Achdou, Mathieu Laurière

Abstract: The theory of mean field games aims at studying deterministic or stochastic differential games (Nash equilibria) as the number of agents tends to infinity. Since very few mean field games have explicit or semi-explicit solutions, numerical simulations play a crucial role in obtaining quantitative information from this class of models. They may lead to systems of evolutive partial differential equa… ▽ More The theory of mean field games aims at studying deterministic or stochastic differential games (Nash equilibria) as the number of agents tends to infinity. Since very few mean field games have explicit or semi-explicit solutions, numerical simulations play a crucial role in obtaining quantitative information from this class of models. They may lead to systems of evolutive partial differential equations coupling a backward Bellman equation and a forward Fokker-Planck equation. In the present survey, we focus on such systems. The forward-backward structure is an important feature of this system, which makes it necessary to design unusual strategies for mathematical analysis and numerical approximation. In this survey, several aspects of a finite difference method used to approximate the previously mentioned system of PDEs are discussed, including convergence, variational aspects and algorithms for solving the resulting systems of nonlinear equations. Finally, we discuss in details two applications of mean field games to the study of crowd motion and to macroeconomics, a comparison with mean field type control, and present numerical simulations. △ Less

Submitted 9 March, 2020; originally announced March 2020.

Comments: This work is part of the C.I.M.E. Lecture Notes to be published in the Springer series "Lecture Notes in Mathematics", corresponding to the course taught by the first author in Cetraro in June 2019

Large Banking Systems with Default and Recovery: A Mean Field Game Model

Authors: Romuald Élie, Tomoyuki Ichiba, Mathieu Laurière

Abstract: We consider a mean-field model for large banking systems, which takes into account default and recovery of the institutions. Building on models used for groups of interacting neurons, we first study a McKean-Vlasov dynamics and its evolutionary Fokker-Planck equation in which the mean-field interactions occur through a mean-reverting term and through a hitting time corresponding to a default level… ▽ More We consider a mean-field model for large banking systems, which takes into account default and recovery of the institutions. Building on models used for groups of interacting neurons, we first study a McKean-Vlasov dynamics and its evolutionary Fokker-Planck equation in which the mean-field interactions occur through a mean-reverting term and through a hitting time corresponding to a default level. The latter feature reflects the impact of a financial institution's default on the global distribution of reserves in the banking system. The systemic risk problem of financial institutions is understood as a blow-up phenomenon of the Fokker-Planck equation. Then, we incorporate in the model an optimization component by letting the institutions control part of their dynamics in order to minimize their expected risk. Phrasing this optimization problem as a mean-field game, we provide an explicit solution in a special case and, in the general case, we report numerical experiments based on a finite difference scheme. △ Less

Submitted 28 January, 2020; originally announced January 2020.

Optimal control of conditioned processes with feedback controls

Authors: Yves Achdou, Mathieu Laurière, Pierre-Louis Lions

Abstract: We consider a class of closed loop stochastic optimal control problems in finite time horizon, in which the cost is an expectation conditional on the event that the process has not exited a given bounded domain. An important difficulty is that the probability of the event that conditionates the strategy decays as time grows. The optimality conditions consist of a system of partial differential equ… ▽ More We consider a class of closed loop stochastic optimal control problems in finite time horizon, in which the cost is an expectation conditional on the event that the process has not exited a given bounded domain. An important difficulty is that the probability of the event that conditionates the strategy decays as time grows. The optimality conditions consist of a system of partial differential equations, including a Hamilton-Jacobi-Bellman equation (backward w.r.t. time) and a (forward w.r.t. time) Fokker-Planck equation for the law of the conditioned process. The two equations are supplemented with Dirichlet conditions. Next, we discuss the asymptotic behavior as the time horizon tends to $+\infty$. This leads to a new kind of optimal control problem driven by an eigenvalue problem related to a continuity equation with Dirichlet conditions on the boundary. We prove existence for the latter. We also propose numerical methods and supplement the various theoretical aspects with numerical simulations. △ Less

Submitted 18 December, 2019; originally announced December 2019.

Stochastic Graphon Games: I. The Static Case

Authors: Rene Carmona, Daniel Cooney, Christy Graves, Mathieu Lauriere

Abstract: We consider static finite-player network games and their continuum analogs, graphon games. Existence and uniqueness results are provided, as well as convergence of the finite-player network game optimal strategy profiles to their analogs for the graphon games. We also show that equilibrium strategy profiles of a graphon game provide approximate Nash equilibria for the finite-player games. Connecti… ▽ More We consider static finite-player network games and their continuum analogs, graphon games. Existence and uniqueness results are provided, as well as convergence of the finite-player network game optimal strategy profiles to their analogs for the graphon games. We also show that equilibrium strategy profiles of a graphon game provide approximate Nash equilibria for the finite-player games. Connections with mean field games and central planner optimization problems are discussed. Motivating applications are presented and explicit computations of their Nash equilibria and social optimal strategies are provided. △ Less

Submitted 24 November, 2019; originally announced November 2019.

Comments: 5 figures

Backward propagation of chaos

Authors: Mathieu Laurière, Ludovic Tangpi

Abstract: This paper develops a theory of propagation of chaos for a system of weakly interacting particles whose terminal configuration is fixed as opposed to the initial configuration as customary. Such systems are modeled by backward stochastic differential equations. Under standard assumptions on the coefficients of the equations, we prove propagation of chaos results and quantitative estimates on the r… ▽ More This paper develops a theory of propagation of chaos for a system of weakly interacting particles whose terminal configuration is fixed as opposed to the initial configuration as customary. Such systems are modeled by backward stochastic differential equations. Under standard assumptions on the coefficients of the equations, we prove propagation of chaos results and quantitative estimates on the rate of convergence in Wasserstein distance of the empirical measure of the interacting system to the law of a McKean-Vlasov type equation. These results are accompanied by non-asymptotic concentration inequalities. As an application, we derive rate of convergence results for solutions of second order semilinear partial differential equations to the solution of a partial differential written on an infinite dimensional space. △ Less

Submitted 15 November, 2019; originally announced November 2019.

Model-Free Mean-Field Reinforcement Learning: Mean-Field MDP and Mean-Field Q-Learning

Authors: René Carmona, Mathieu Laurière, Zongjun Tan

Abstract: We study infinite horizon discounted Mean Field Control (MFC) problems with common noise through the lens of Mean Field Markov Decision Processes (MFMDP). We allow the agents to use actions that are randomized not only at the individual level but also at the level of the population. This common randomization allows us to establish connections between both closed-loop and open-loop policies for MFC… ▽ More We study infinite horizon discounted Mean Field Control (MFC) problems with common noise through the lens of Mean Field Markov Decision Processes (MFMDP). We allow the agents to use actions that are randomized not only at the individual level but also at the level of the population. This common randomization allows us to establish connections between both closed-loop and open-loop policies for MFC and Markov policies for the MFMDP. In particular, we show that there exists an optimal closed-loop policy for the original MFC. Building on this framework and the notion of state-action value function, we then propose reinforcement learning (RL) methods for such problems, by adapting existing tabular and deep RL methods to the mean-field setting. The main difficulty is the treatment of the population state, which is an input of the policy and the value function. We provide convergence guarantees for tabular algorithms based on discretizations of the simplex. Neural network based algorithms are more suitable for continuous spaces and allow us to avoid discretizing the mean field state space. Numerical examples are provided. △ Less

Submitted 13 October, 2021; v1 submitted 28 October, 2019; originally announced October 2019.

Linear-Quadratic Mean-Field Reinforcement Learning: Convergence of Policy Gradient Methods

Authors: René Carmona, Mathieu Laurière, Zongjun Tan

Abstract: We investigate reinforcement learning for mean field control problems in discrete time, which can be viewed as Markov decision processes for a large number of exchangeable agents interacting in a mean field manner. Such problems arise, for instance when a large number of robots communicate through a central unit dispatching the optimal policy computed by minimizing the overall social cost. An appr… ▽ More We investigate reinforcement learning for mean field control problems in discrete time, which can be viewed as Markov decision processes for a large number of exchangeable agents interacting in a mean field manner. Such problems arise, for instance when a large number of robots communicate through a central unit dispatching the optimal policy computed by minimizing the overall social cost. An approximate solution is obtained by learning the optimal policy of a generic agent interacting with the statistical distribution of the states of the other agents. We prove rigorously the convergence of exact and model-free policy gradient methods in a mean-field linear-quadratic setting. We also provide graphical evidence of the convergence based on implementations of our algorithms. △ Less

Submitted 9 October, 2019; originally announced October 2019.

Convergence Analysis of Machine Learning Algorithms for the Numerical Solution of Mean Field Control and Games: II -- The Finite Horizon Case

Authors: René Carmona, Mathieu Laurière

Abstract: We propose two numerical methods for the optimal control of McKean-Vlasov dynamics in finite time horizon. Both methods are based on the introduction of a suitable loss function defined over the parameters of a neural network. This allows the use of machine learning tools, and efficient implementations of stochastic gradient descent in order to perform the optimization. In the first method, the lo… ▽ More We propose two numerical methods for the optimal control of McKean-Vlasov dynamics in finite time horizon. Both methods are based on the introduction of a suitable loss function defined over the parameters of a neural network. This allows the use of machine learning tools, and efficient implementations of stochastic gradient descent in order to perform the optimization. In the first method, the loss function stems directly from the optimal control problem. The second method tackles a generic forward-backward stochastic differential equation system (FBSDE) of McKean-Vlasov type, and relies on suitable reformulation as a mean field control problem. To provide a guarantee on how our numerical schemes approximate the solution of the original mean field control problem, we introduce a new optimization problem, directly amenable to numerical computation, and for which we rigorously provide an error rate. Several numerical examples are provided. Both methods can easily be applied to certain problems with common noise, which is not the case with the existing technology. Furthermore, although the first approach is designed for mean field control problems, the second is more general and can also be applied to the FBSDE arising in the theory of mean field games. △ Less

Submitted 29 March, 2021; v1 submitted 5 August, 2019; originally announced August 2019.

Convergence Analysis of Machine Learning Algorithms for the Numerical Solution of Mean Field Control and Games: I -- The Ergodic Case

Authors: René Carmona, Mathieu Laurière

Abstract: We propose two algorithms for the solution of the optimal control of ergodic McKean-Vlasov dynamics. Both algorithms are based on approximations of the theoretical solutions by neural networks, the latter being characterized by their architecture and a set of parameters. This allows the use of modern machine learning tools, and efficient implementations of stochastic gradient descent.The first alg… ▽ More We propose two algorithms for the solution of the optimal control of ergodic McKean-Vlasov dynamics. Both algorithms are based on approximations of the theoretical solutions by neural networks, the latter being characterized by their architecture and a set of parameters. This allows the use of modern machine learning tools, and efficient implementations of stochastic gradient descent.The first algorithm is based on the idiosyncrasies of the ergodic optimal control problem. We provide a mathematical proof of the convergence of the approximation scheme, and we analyze rigorously the approximation by controlling the different sources of error. The second method is an adaptation of the deep Galerkin method to the system of partial differential equations issued from the optimality condition. We demonstrate the efficiency of these algorithms on several numerical examples, some of them being chosen to show that our algorithms succeed where existing ones failed. We also argue that both methods can easily be applied to problems in dimensions larger than what can be found in the existing literature. Finally, we illustrate the fact that, although the first algorithm is specifically designed for mean field control problems, the second one is more general and can also be applied to the partial differential equation systems arising in the theory of mean field games. △ Less

Submitted 29 March, 2021; v1 submitted 12 July, 2019; originally announced July 2019.

On the Convergence of Model Free Learning in Mean Field Games

Authors: Romuald Elie, Julien Pérolat, Mathieu Laurière, Matthieu Geist, Olivier Pietquin

Abstract: Learning by experience in Multi-Agent Systems (MAS) is a difficult and exciting task, due to the lack of stationarity of the environment, whose dynamics evolves as the population learns. In order to design scalable algorithms for systems with a large population of interacting agents (e.g. swarms), this paper focuses on Mean Field MAS, where the number of agents is asymptotically infinite. Recently… ▽ More Learning by experience in Multi-Agent Systems (MAS) is a difficult and exciting task, due to the lack of stationarity of the environment, whose dynamics evolves as the population learns. In order to design scalable algorithms for systems with a large population of interacting agents (e.g. swarms), this paper focuses on Mean Field MAS, where the number of agents is asymptotically infinite. Recently, a very active burgeoning field studies the effects of diverse reinforcement learning algorithms for agents with no prior information on a stationary Mean Field Game (MFG) and learn their policy through repeated experience. We adopt a high perspective on this problem and analyze in full generality the convergence of a fictitious iterative scheme using any single agent learning algorithm at each step. We quantify the quality of the computed approximate Nash equilibrium, in terms of the accumulated errors arising at each learning iteration step. Notably, we show for the first time convergence of model free learning algorithms towards non-stationary MFG equilibria, relying only on classical assumptions on the MFG dynamics. We illustrate our theoretical results with a numerical experiment in a continuous action-space environment, where the approximate best response of the iterative fictitious play scheme is computed with a deep RL algorithm. △ Less

Submitted 20 February, 2020; v1 submitted 4 July, 2019; originally announced July 2019.

Journal ref: AAAI 2020 conference proceedings