Reinforcement Learning for Unified Allocation and Patrolling in Signaling Games with Uncertainty
Authors: 
Aravind Venugopal, Elizabeth Bondi, Harshavardhan Kamarthi, Keval Dholakia, Balaraman Ravindran, Milind TambeAuthors Info & Claims
Aravind Venugopal, Elizabeth Bondi, Harshavardhan Kamarthi, Keval Dholakia, Balaraman Ravindran, Milind TambeAuthors Info & ClaimsPages 1353 - 1361
Abstract
Green Security Games (GSGs) have been successfully used in the protection of valuable resources such as fisheries, forests, and wildlife. Real-world deployment involves both resource allocation and subsequent coordinated patrolling with communication in the presence real-time, uncertain information. Previous game models do not address both of these stages simultaneously. Furthermore, adopting existing solution strategies is difficult since they do not scale well for larger, more complex variants of the game models. We propose a novel GSG model to address these challenges. We also present a novel algorithm, CombSGPO, to compute a defender strategy for this game model. CombSGPO performs policy search over a multidimensional, discrete action space to compute an allocation strategy that is best suited to a best-response patrolling strategy for the defender, learnt by training a multi-agent Deep Q-Network. We show via experiments that CombSGPO converges to better strategies and is more scalable than comparable approaches. From a detailed analysis of the coordination and signaling behavior learnt by CombSGPO, we find that strategic signaling emerges in the final learnt strategy.
References
[1]
Anjon Basak, Fei Fang, Thanh Hong Nguyen, and Christopher Kiekintveld. 2016. Combining graph contraction and strategy generation for green security games. In International Conference on Decision and Game Theory for Security. Springer, 251--271.
[2]
Nicola Basilico, Giuseppe De Nittis, and Nicola Gatti. 2015. A security game model for environment protection in the presence of an alarm system. In International Conference on Decision and Game Theory for Security. Springer, 192--207.
[3]
Elizabeth Bondi, Fei Fang, Mark Hamilton, Debarun Kar, Donnabell Dmello, Jongmoo Choi, Robert Hannaford, Arvind Iyer, Lucas Joppa, Milind Tambe, et al. 2018. Spot poachers in action: Augmenting conservation drones with automatic detection in near real time. In IAAI. 7741--7746.
[4]
Elizabeth Bondi, Hoon Oh, Haifeng Xu, Fei Fang, Bistra Dilkina, and Milind Tambe. 2020. To Signal or Not To Signal: Exploiting Uncertain Real-Time Information in Signaling Games for Security and Sustainability. In AAAI. 1369--1377.
[5]
Yash Chandak, Georgios Theocharous, James Kostas, Scott Jordan, and Philip S Thomas. 2019. Learning action representations for reinforcement learning. arXiv preprint arXiv:1902.00183 (2019).
[6]
Fei Fang, Thanh Hong Nguyen, Rob Pickles, Wai Y Lam, Gopalasamy R Clements, Bo An, Amandeep Singh, Milind Tambe, Andrew Lemieux, et al. 2016. Deploying PAWS: Field Optimization of the Protection Assistant for Wildlife Security. In AAAI, Vol. 16. 3966--3973.
[7]
Fei Fang, Peter Stone, and Milind Tambe. 2015. When Security Games Go Green: Designing Defender Strategies to Prevent Poaching and Illegal Fishing. In IJCAI. 2589--2595.
[8]
Drew Fudenberg, Fudenberg Drew, David K Levine, and David K Levine. 1998. The theory of learning in games. Vol. 2. MIT press.
[9]
Eric A Hansen, Daniel S Bernstein, and Shlomo Zilberstein. 2004. Dynamic programming for partially observable stochastic games. In AAAI, Vol. 4. 709--715.
[10]
Johannes Heinrich and David Silver. 2016. Deep reinforcement learning from self-play in imperfect-information games. arXiv preprint arXiv:1603.01121 (2016).
[11]
Nitin Kamra, Umang Gupta, Fei Fang, Yan Liu, and Milind Tambe. 2018. Policy Learning for Continuous Space Security Games Using Neural Networks. In AAAI. 1103--1112.
[12]
Nitin Kamra, Umang Gupta, Kai Wang, Fei Fang, Yan Liu, and Milind Tambe. 2019. DeepFP for Finding Nash Equilibrium in Continuous Action Spaces. In International Conference on Decision and Game Theory for Security. Springer, 238--258.
[13]
Richard Klima, Karl Tuyls, and Frans A Oliehoek. 2018. Model-based reinforcement learning under periodical observability. In 2018 AAAI Spring Symposium Series .
[14]
Dmytro Korzhyk, Zhengyu Yin, Christopher Kiekintveld, Vincent Conitzer, and Milind Tambe. 2011. Stackelberg vs. Nash in security games: An extended investigation of interchangeability, equivalence, and uniqueness. Journal of Artificial Intelligence Research, Vol. 41 (2011), 297--327.
[15]
Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. 2017. A unified game-theoretic approach to multiagent reinforcement learning. In Advances in neural information processing systems. 4190--4203.
[16]
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. 2017. Multi-agent actor-critic for mixed cooperative-competitive environments. arXiv preprint arXiv:1706.02275 (2017).
[17]
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013).
[18]
James Pita, Manish Jain, C Western, P Paruchuri, J Marecki, M Tambe, F Ordonez, and Sarit Kraus. 2009. Armor software: A game theoretic approach to airport security. Protecting Airline Passengers in the Age of Terrorism (2009), 163.
[19]
Manish Prajapat, Kamyar Azizzadenesheli, Alexander Liniger, Yisong Yue, and Anima Anandkumar. 2020. Competitive Policy Optimization. arXiv preprint arXiv:2006.10611 (2020).
[20]
David V Pynadath and Milind Tambe. 2002. The communicative multiagent team decision problem: Analyzing teamwork theories and models. Journal of artificial intelligence research, Vol. 16 (2002), 389--423.
[21]
Eric Shieh, Albert Jiang, Amulya Yadav, Pradeep Reddy VARAKANTHAM, and Milind Tambe. 2014. Unleashing dec-mdps in security games: Enabling effective defender teamwork. (2014).
[22]
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. 2000. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems. 1057--1063.
[23]
Milind Tambe. 2011. Security and game theory: algorithms, deployed systems, lessons learned .Cambridge university press.
[24]
Jason Tsai, Shyamsunder Rathi, Christopher Kiekintveld, Fernando Ordonez, and Milind Tambe. 2009. IRIS-a tool for strategic security allocation in transportation networks. AAMAS (Industry Track) (2009), 37--44.
[25]
Hado Van Hasselt, Arthur Guez, and David Silver. 2015. Deep reinforcement learning with double q-learning. arXiv preprint arXiv:1509.06461 (2015).
[26]
Yufei Wang, Zheyuan Ryan Shi, Lantao Yu, Yi Wu, Rohit Singh, Lucas Joppa, and Fei Fang. 2019. Deep reinforcement learning for green security games with real-time information. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 1401--1408.
[27]
Haifeng Xu, Benjamin Ford, Fei Fang, Bistra Dilkina, Andrew Plumptre, Milind Tambe, Margaret Driciru, Fred Wanyama, Aggrey Rwetsiba, Mustapha Nsubaga, et al. 2017. Optimal patrol planning for green security games with black-box attackers. In International Conference on Decision and Game Theory for Security. Springer, 458--477.
[28]
Haifeng Xu, Kai Wang, Phebe Vayanos, and Milind Tambe. 2018. Strategic coordination of human patrollers and mobile sensors with signaling for security games. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32.
[29]
Zhengyu Yin, Albert Xin Jiang, Milind Tambe, Christopher Kiekintveld, Kevin Leyton-Brown, Tuomas Sandholm, and John P Sullivan. 2012. TRUSTS: Scheduling randomized patrols for fare inspection in transit systems using game theory. AI magazine, Vol. 33, 4 (2012), 59--59.
Index Terms
- Reinforcement Learning for Unified Allocation and Patrolling in Signaling Games with Uncertainty
Recommendations
Reinforcement Learning in Multi-agent Games: Open AI Gym Diplomacy Environment
Progress in Artificial IntelligenceAbstractReinforcement learning has been successfully applied to adversarial games, exhibiting its potential. However, most real-life scenarios also involve cooperation, in addition to competition. Using reinforcement learning in multi-agent cooperative ...
Reinforcement Learning for N-player Games: The Importance of Final Adaptation
Bioinspired Optimization Methods and Their ApplicationsAbstractThis paper covers n-tuple-based reinforcement learning (RL) algorithms for games. We present a new algorithm for temporal difference (TD) learning which works seamlessly on various games with arbitrary number of players. This is achieved by taking ...
Comments
Information & Contributors
Information
Published In
May 2021
1899 pages
ISBN:9781450383073
- General Chairs:
- Frank Dignum,
- Alessio Lomuscio,
- Program Chairs:
- Ulle Endriss,
- Ann Nowé
Sponsors
Publisher
International Foundation for Autonomous Agents and Multiagent Systems
Richland, SC
Publication History
Published: 03 May 2021
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
Conference
AAMAS '21
Sponsor:
AAMAS '21: 20th International Conference on Autonomous Agents and Multiagent Systems
May 3 - 7, 2021
Virtual Event, United Kingdom
Acceptance Rates
Overall Acceptance Rate 1,155 of 5,036 submissions, 23%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 227Total Downloads
- Downloads (Last 12 months)67
- Downloads (Last 6 weeks)15
Reflects downloads up to 09 Nov 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in