Computer Science > Machine Learning
[Submitted on 16 Apr 2018 (v1), revised 27 Sep 2018 (this version, v2), latest version 20 Feb 2020 (v3)]
Title:On the Convergence of Gradient-Based Learning in Continuous Games
View PDFAbstract:We study the limiting behavior of competitive agents employing gradient-based learning algorithms through the lens of dynamical systems theory. Specifically, we introduce a general framework for competitive gradient-based learning that allows us to analyze a wide breadth of learning algorithms including policy gradient reinforcement learning, gradient based bandits, and certain online convex optimization algorithms. We show that for both potential games and general-sum games, when agents employ gradient-based learning algorithms, they will avoid a non-negligible subset of the local Nash equilibria. This is a strongly negative result for gradient-based learning in games. Our framework also sheds light on the issue of convergence to non-Nash strategies in general-sum and zero-sum games which have no relevance to the underlying game, and arise solely due to the choice of algorithm. The existence and frequency of strategies may explain some of the difficulties encountered when using gradient descent in zero-sum games (e.g. to train generative adversarial networks). Finally, we introduce a new class of games, Morse-Smale games, for which the gradient dynamics correspond to gradient-like flows. This class encompasses a large set of commonly encountered games. For Morse-Smale games, we show that competitive gradient-based learning converges to either limit cycles, Nash equilibria, or non-Nash fixed points almost surely. To reinforce our theoretical contributions, we provide empirical results that highlight the frequency of Nash equilibria that are almost surely avoided by policy gradient in linear quadratic games. Indeed, we present empirical results that show that policy gradient almost surely avoids the unique global Nash equilibrium in one out of five randomly sampled linear quadratic games.
Submission history
From: Eric Mazumdar [view email][v1] Mon, 16 Apr 2018 01:14:17 UTC (2,655 KB)
[v2] Thu, 27 Sep 2018 03:54:44 UTC (3,119 KB)
[v3] Thu, 20 Feb 2020 18:26:35 UTC (634 KB)
Current browse context:
cs.LG
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
IArxiv Recommender
(What is IArxiv?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.