Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.5555/3306127.3332004acmconferencesArticle/Chapter ViewAbstractPublication PagesaamasConference Proceedingsconference-collections
research-article

Deep Fictitious Play for Games with Continuous Action Spaces

Published: 08 May 2019 Publication History

Abstract

Fictitious play has been a classic algorithm to solve two-player adversarial games with discrete action spaces. In this work we develop an approximate extension of fictitious play to two-player games with high-dimensional continuous action spaces. We use generative neural networks to approximate players' best responses while also learning a differentiable approximate model to the players' rewards given their actions. Both these networks are trained jointly with gradient-based optimization to emulate fictitious play. We explore our approach in zero-sum games, non zero-sum games and security game domains.

References

[1]
Soheil Behnezhad, Mahsa Derakhshan, Mohammadtaghi Hajiaghayi, and Saeed Seddighin. 2018. Spatio-Temporal Games Beyond One Dimension. In Proceedings of the 2018 ACM Conference on Economics and Computation. 411--428.
[2]
Soheil Behnezhad, Mahsa Derakhshan, MohammadTaghi Hajiaghayi, and Aleksandrs Slivkins. 2017. A Polynomial Time Algorithm for Spatio-Temporal Security Games. In Proceedings of the 2017 ACM Conference on Economics and Computation. 697--714.
[3]
Vincent Conitzer. 2009. Approximation guarantees for fictitious play. In 47th Annual Allerton Conference on Communication, Control, and Computing. IEEE, 636--643.
[4]
Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Michael Rabbat, and Joelle Pineau. 2018. TarMAC: Targeted Multi-Agent Communication. arXiv preprint arXiv:1810.11187 (2018).
[5]
Fei Fang, Albert Xin Jiang, and Milind Tambe. 2013. Optimal patrol strategy for protecting moving targets with multiple mobile resources. In AAMAS. 957--964.
[6]
Jiarui Gan, Bo An, Yevgeniy Vorobeychik, and Brian Gauch. 2017. Security Games on a Plane. In AAAI. 530--536.
[7]
William Haskell, Debarun Kar, Fei Fang, Milind Tambe, Sam Cheung, and Elizabeth Denicola. 2014. Robust protection of fisheries with COmPASS. In IAAI .
[8]
Johannes Heinrich, Marc Lanctot, and David Silver. 2015. Fictitious Self-Play in Extensive-Form Games. In ICML. 805--813.
[9]
Johannes Heinrich and David Silver. 2016. Deep Reinforcement Learning from Self-Play in Imperfect-Information Games. CoRR, Vol. abs/1603.01121 (2016).
[10]
Josef Hofbauer and William H Sandholm. 2002. On the global convergence of stochastic fictitious play. Econometrica, Vol. 70, 6 (2002), 2265--2294.
[11]
Matthew P. Johnson, Fei Fang, and Milind Tambe. 2012. Patrol Strategies to Maximize Pristine Forest Area. In AAAI .
[12]
Nitin Kamra, Fei Fang, Debarun Kar, Yan Liu, and Milind Tambe. 2017. Handling continuous space security games with neural networks. In IWAISe: First International Workshop on Artificial Intelligence in Security .
[13]
Nitin Kamra, Umang Gupta, Fei Fang, Yan Liu, and Milind Tambe. 2018. Policy Learning for Continuous Space Security Games using Neural Networks. In AAAI .
[14]
Debarun Kar, Fei Fang, Francesco Delle Fave, Nicole Sintov, and Milind Tambe. 2015. "A Game of Thrones": When Human Behavior Models Compete in Repeated Stackelberg Security Games. In AAMAS .
[15]
Taesup Kim and Yoshua Bengio. 2016. Deep directed generative models with energy-based probability estimation. arXiv preprint arXiv:1606.03439 (2016).
[16]
Vijay Krishna and Tomas Sjöström. 1998. On the convergence of fictitious play. Mathematics of Operations Research, Vol. 23, 2 (1998), 479--511.
[17]
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. 2017. Multi-agent actor-critic for mixed cooperative-competitive environments. In Advances in Neural Information Processing Systems. 6379--6390.
[18]
Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P How, and John Vian. 2017. Deep decentralized multi-task multi-agent reinforcement learning under partial observability. arXiv preprint arXiv:1703.06182 (2017).
[19]
S. Perkins and D.S. Leslie. 2014. Stochastic fictitious play with continuous action sets. Journal of Economic Theory, Vol. 152 (2014), 179 -- 213.
[20]
Jeff S Shamma and Gürdal Arslan. 2004. Unified convergence proofs of continuous-time fictitious play. IEEE Trans. Automat. Control, Vol. 49, 7 (2004), 1137--1141.
[21]
Binru Wang, Yuan Zhang, and Sheng Zhong. 2017. On Repeated Stackelberg Security Game with the Cooperative Human Behavior Model for Wildlife Protection. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems (AAMAS '17). 1751--1753.
[22]
Rong Yang, Benjamin Ford, Milind Tambe, and Andrew Lemieux. 2014. Adaptive Resource Allocation for Wildlife Protection against Illegal Poachers. In AAMAS.
[23]
Yue Yin, Bo An, and Manish Jain. 2014. Game-theoretic Resource Allocation for Protecting Large Public Events. In AAAI. 826--833.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
AAMAS '19: Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems
May 2019
2518 pages
ISBN:9781450363099

Sponsors

Publisher

International Foundation for Autonomous Agents and Multiagent Systems

Richland, SC

Publication History

Published: 08 May 2019

Check for updates

Author Tags

  1. Nash equilibrium
  2. deep learning
  3. fictitious play
  4. multi-agent learning
  5. stackelberg security games
  6. two-player games

Qualifiers

  • Research-article

Conference

AAMAS '19
Sponsor:

Acceptance Rates

AAMAS '19 Paper Acceptance Rate 193 of 793 submissions, 24%;
Overall Acceptance Rate 1,155 of 5,036 submissions, 23%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 94
    Total Downloads
  • Downloads (Last 12 months)1
  • Downloads (Last 6 weeks)0
Reflects downloads up to 05 Jan 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media