Abstract
In a multi-agent game, the complexity of the environment increases exponentially as the number of agents increases. Learning becomes difficult when there are so many agents. Mean field multi-agent reinforcement learning (MFRL) uses the average action of the neighbors to increase the input of the value network, which can be applied in the environment with hundreds of agents. However, inefficient exploration and slow convergence speed limit the performance of the algorithm. In this article, we propose a new Knowledge-Guided Reinforcement Learning (KG-RL) method, which can be divided into rule-mix and plan-extend. We use the rule-mix to encode knowledge into plans which can reduce redundant information and invalid actions in the state. And the plan-extend can combine the result of rule-mix with reinforcement learning to achieve more efficient joint exploration. Through experiments in Magent environment, we prove that the win rate of our proposed KG-RL is 22% higher than that of knowledge-based decision tree and 39% higher than that of MFRL. Thus, the KG-RL can perform well in massive battle games due to its high exploration efficiency and fast convergence.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
References
Bougie, N., Ichise, R.: Deep reinforcement learning boosted by external knowledge. In: Proceedings of the 33rd Annual ACM Symposium on Applied Computing, pp. 331–338 (2018)
Chen, X., Huang, C., Yao, L., Wang, X., Zhang, W., et al.: Knowledge-guided deep reinforcement learning for interactive recommendation. In: 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2020)
Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K.O., Clune, J.: First return, then explore. Nature 590(7847), 580–586 (2021)
Elo, A.E.: The rating of chessplayers, past and present. Arco Pub. (1978)
Han, X., Tang, H., Li, Y., Kou, G., Liu, L.: Improving multi-agent reinforcement learning with imperfect human knowledge. In: International Conference on Artificial Neural Networks, pp. 369–380. Springer (2020). https://doi.org/10.1007/978-3-030-61616-8_30
Hernandez-Leal, P., Kartal, B., Taylor, M.E.: A survey and critique of multiagent deep reinforcement learning
Jaderberg, M., Czarnecki, W., Dunning, I., Marris, L., Lever, G., Castaneda, A., et al.: Human-level performance in first-person multiplayer games with population-based deep reinforcement learning. arXiv:1807.01281 (2018)
Leonetti, M., Iocchi, L., Stone, P.: A synthesis of automated planning and reinforcement learning for efficient, robust decision-making. Artif. Intell. 241, 103–130 (2016)
Lillicrap, T.P., et al.: Continuous control with deep reinforcement learning. arXiv:1509.02971 (2015)
Liu, Y., Wang, W., Hu, Y., Hao, J., Chen, X., Gao, Y.: Multi-agent game abstraction via graph attention neural network. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 7211–7218 (2020)
Long, Q., Zhou, Z., Gupta, A., Fang, F., Wu, Y., Wang, X.: Evolutionary population curriculum for scaling multi-agent reinforcement learning. arXiv:2003.10423 (2020)
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., Mordatch, I.: Multi-agent actor-critic for mixed cooperative-competitive environments. NIPS (2017)
Mao, H., et al.: Neighborhood cognition consistent multi-agent reinforcement learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 7219–7226 (2020)
Mnih, V., et al.: Asynchronous methods for deep reinforcement learning. In: International Conference on Machine Learning, pp. 1928–1937. PMLR (2016)
Rashid, T., Samvelyan, M., Schroeder, C., Farquhar, G., Foerster, J., Whiteson, S.: Qmix: monotonic value function factorisation for deep multi-agent reinforcement learning. In: International Conference on Machine Learning, pp. 4295–4304. PMLR (2018)
Silva, A., Gombolay, M.: Neural-encoding human experts’ domain knowledge to warm start reinforcement learning. arXiv:1902.06007 (2019)
Tan, M.: Multi-agent reinforcement learning: Independent vs. cooperative agents. In: Proceedings of the Tenth International Conference on Machine Learning. pp. 330–337 (1993)
Tang, Z., et al.: Discovering diverse multi-agent strategic behavior via reward randomization. arXiv:2103.04564 (2021)
Wang, P., Fan, Y., Xia, L., Zhao, W.X., Niu, S., Huang, J.: Kerl: a knowledge-guided reinforcement learning model for sequential recommendation. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 209–218 (2020)
Xie, L., et al.: Learning with stochastic guidance for navigation. arXiv:1811.10756 (2018)
Yang, Y., Rui, L., Li, M., Ming, Z., Wang, J.: Mean field multi-agent reinforcement learning (2018)
Zheng, L., et al.: Magent: A many-agent reinforcement learning platform for artificial collective intelligence. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018)
Zhou, M., Liu, Z., Sui, P., Li, Y., Chung, Y.Y.: Learning implicit credit assignment for cooperative multi-agent reinforcement learning (2020)
Acknowledgment
This work was partly supported by both National Key Research and Development Program of China (Grant No.2019AAA0104800) and NSFC under grant No. 91648204.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhou, S., Ren, W., Ren, X., Mi, X., Yi, X. (2021). KG-RL: A Knowledge-Guided Reinforcement Learning for Massive Battle Games. In: Pham, D.N., Theeramunkong, T., Governatori, G., Liu, F. (eds) PRICAI 2021: Trends in Artificial Intelligence. PRICAI 2021. Lecture Notes in Computer Science(), vol 13033. Springer, Cham. https://doi.org/10.1007/978-3-030-89370-5_7
Download citation
DOI: https://doi.org/10.1007/978-3-030-89370-5_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-89369-9
Online ISBN: 978-3-030-89370-5
eBook Packages: Computer ScienceComputer Science (R0)