Investigating the Performance of Various Deep Neural Networks-based Approaches Designed to Identify Game Events in Gameplay Footage
Authors: 
Matheus Prado Prandini Faria, Etienne Silva Julia, Marcelo Zanchetta do Nascimento, Rita Maria Silva JuliaAuthors Info & Claims
Matheus Prado Prandini Faria, Etienne Silva Julia, Marcelo Zanchetta do Nascimento, Rita Maria Silva JuliaAuthors Info & ClaimsArticle No.: 8, Pages 1 - 17
Abstract
Video games, in addition to representing an extremely relevant field of entertainment and market, have been widely used as a case study in artificial intelligence for representing a problem with a high degree of complexity. In such studies, the investigation of approaches that endow player agents with the ability to retrieve relevant information from game scenes stands out, since such information can be very useful to improve their learning ability. This work proposes and analyses new deep learning-based models to identify game events occurring in Super Mario Bros gameplay footage. The architecture of each model is composed of a feature extractor convolutional neural network (CNN) and a classifier neural network (NN). The extracting CNN aims to produce a feature-based representation for game scenes and submit it to the classifier, so that the latter can identify the game event present in each scene. The models differ from each other according to the following elements: the type of the CNN; the type of the NN classifier; and the type of the game scene representation at the CNN input, being either single frames, or chunks, which are n-sequential frames (in this paper 6 frames were used per chunk) grouped into a single input. The main contribution of this article is to demonstrate the greater performance reached by the models which combines the chunk representation for the game scenes with the resources of the classifier recurrent neural networks (RNN).
References
[1]
N. Aloysius and M. Geetha. 2017. A review on deep convolutional neural networks. In 2017 International Conference on Communication and Signal Processing (ICCSP). 0588--0592. https://doi.org/10.1109/ICCSP.2017.8286426
[2]
Leonard A. Annetta. 2008. Video Games in Education: Why They Should Be Used and How They Are Being Used. Theory Into Practice (2008). https://doi.org/10.1080/00405840802153940
[3]
Elizabeth Boyle, Thomas M. Connolly, and Thomas Hainey. 2011. The role of psychology in understanding the impact of computer games. Entertainment Computing 2, 2 (2011), 69--74. https://doi.org/10.1016/j.entcom.2010.12.002 Serious Games Development and Applications.
[4]
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. arXiv:1406.1078 [cs.CL]
[5]
Jiyang Gao, Zhenheng Yang, and Ram Nevatia. 2017. RED: Reinforced Encoder-Decoder Networks for Action Anticipation. arXiv:1707.04818 http://arxiv.org/abs/1707.04818
[6]
Roeland De Geest, Efstratios Gavves, Amir Ghodrati, Zhenyang Li, Cees Snoek, and Tinne Tuytelaars. 2016a. Online Action Detection. arXiv:1604.06506 [cs.CV]
[7]
Roeland De Geest, Efstratios Gavves, Amir Ghodrati, Zhenyang Li, Cees Snoek, and Tinne Tuytelaars. 2016b. Online Action Detection. CoRR abs/1604.06506. arXiv:1604.06506 http://arxiv.org/abs/1604.06506
[8]
Global Data. 2021. Video games market set to become a 300bn-plus industry by 2025. https://www.globaldata.com/video-games-market-set-to-become-a-300bn-plus-industry-by-2025.
[9]
Matthew Guzdial, Boyang Li, and Mark O. Riedl. 2017. Game Engine Learning from Video. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (Melbourne, Australia) (IJCAI'17). AAAI Press, 3707--3713.
[10]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[11]
Sepp Hochreiter. 1998. The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems 6 (04 1998), 107--116. https://doi.org/10.1142/S0218488598000094
[12]
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation 9, 8 (1997), 1735--1780. https://doi.org/10.1162/neco.1997.9.8.1735 arXiv:https://doi.org/10.1162/neco.1997.9.8.1735
[13]
J J Hopfield. 1982. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the National Academy of Sciences 79 (1982), 2554--2558. https://doi.org/10.1073/pnas.79.8.2554
[14]
V. Janarthanan. 2012. Serious Video Games: Games for Education and Health. In 2012 Ninth International Conference on Information Technology - New Generations. https://doi.org/10.1109/ITNG.2012.79
[15]
Michael I. Jordan. 1997. Chapter 25 - Serial Order: A Parallel Distributed Processing Approach. 121 (1997), 471--495. https://doi.org/10.1016/S0166-4115(97)80111-2
[16]
S. Karakovskiy and J. Togelius. 2012. The Mario AI Benchmark and Competitions. IEEE Transactions on Computational Intelligence and AI in Games 4, 1 (2012), 55--67. https://doi.org/10.1109/TCIAIG.2012.2188528
[17]
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. 2014. Large-Scale Video Classification with Convolutional Neural Networks. In 2014 IEEE Conference on Computer Vision and Pattern Recognition. 1725--1732. https://doi.org/10.1109/CVPR.2014.223
[18]
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
[19]
Michail D. Kozlov and Mark K. Johansen. 2010. Real Behavior in Virtual Environments: Psychology Experiments in a Simple Virtual-Reality Paradigm Using Video Games. Cyberpsychology, Behavior, and Social Networking (2010). https://doi.org/10.1089/cyber.2009.0310
[20]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 1097--1105.
[21]
Kangwook Lee, Hoon Kim, and Changho Suh. 2017. Crash To Not Crash: Playing Video Games To Predict Vehicle Collisions. In ICML 2017.
[22]
Zijin Luo, Matthew Guzdial, Nicholas Liao, and Mark Riedl. 2018. Player Experience Extraction from Gameplay Video. CoRR abs/1809.06201 (2018). arXiv:1809.06201
[23]
Zijin Luo, Matthew Guzdial, and Mark Riedl. 2019. Making CNNs for Video Parsing Accessible. CoRR abs/1906.11877 (2019). arXiv:1906.11877
[24]
M. Ravanbakhsh, M. Nabi, E. Sangineto, L. Marcenaro, C. Regazzoni, and N. Sebe. 2017. Abnormal event detection in videos using generative adversarial nets. In 2017 IEEE International Conference on Image Processing (ICIP). 1577--1581. https://doi.org/10.1109/ICIP.2017.8296547
[25]
Johanna Roettl and Ralf Terlutter. 2018. The same video game in 2D, 3D or virtual reality - How does technology impact game evaluation and brand placements? PLOS ONE 13, 7 (07 2018), 1--24. https://doi.org/10.1371/journal.pone.0200724
[26]
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018. Inverted Residuals and Linear Bottlenecks: Mobile Networks for Classification, Detection and Segmentation. CoRR abs/1801.04381 (2018). arXiv:1801.04381
[27]
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. 2018. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science 362, 6419 (2018), 1140--1144. https://doi.org/10.1126/science.aar6404
[28]
Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv 1409.1556 (09 2014).
[29]
Khurram Soomro, Amir Zamir, and Mubarak Shah. 2012. UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild. CoRR (12 2012).
[30]
Adam Summerville, Sam Snodgrass, Matthew Guzdial, Christoffer Holmgård, Amy K. Hoover, Aaron Isaksen, Andy Nealen, and Julian Togelius. 2017. Procedural Content Generation via Machine Learning (PCGML). CoRR abs/1702.00539 (2017). arXiv:1702.00539 http://arxiv.org/abs/1702.00539
[31]
Jeremy Heng Meng Wong and Mark John Francis Gales. 2016. Sequence Student-Teacher Training of Deep Neural Networks. In INTERSPEECH. https://doi.org/10.21437/Interspeech.2016-911
[32]
Mingze Xu, Mingfei Gao, Yi-Ting Chen, Larry S. Davis, and David J. Crandall. 2018. Temporal Recurrent Networks for Online Action Detection. arXiv:1811.07391 http://arxiv.org/abs/1811.07391
[33]
Manzhu Yu, Myra Bambacus, Guido Cervone, Keith Clarke, Daniel Duffy, Qunying Huang, Jing Li, Wenwen Li, Zhenlong Li, Qian Liu, Bernd Resch, Jingchao Yang, and Chaowei Yang. 2020. Spatiotemporal event detection: a review. International Journal of Digital Earth 13, 12 (2020), 1339--1365. https://doi.org/10.1080/17538947.2020.1738569 arXiv:https://doi.org/10.1080/17538947.2020.1738569
Index Terms
- Investigating the Performance of Various Deep Neural Networks-based Approaches Designed to Identify Game Events in Gameplay Footage
Recommendations
Fruit type classification using deep learning and feature fusion
Highlights- Proposed used deep learning applications to classify the fruits based on optimal features.
- CNN employed to extract the optimal features.
- RNN employed to label the optimal features.
- LSTM employed to classify the fruits based on ...
AbstractMachine and deep learning applications play a dominant role in the current scenario in the agriculture sector. To date, the classification of fruits using image features has attained the researcher’s attraction very much from the last few years. ...
Deep visual tracking
The first comprehensive survey on deep-learning-based trackers.Review existing deep visual trackers from three different perspectives.Large-scale benchmark evaluations of deep visual trackers.Summarize cutting-edge research works and discuss future ...
Synthesizing Game Levels for Collaborative Gameplay in a Shared Virtual Environment
We developed a method to synthesize game levels that accounts for the degree of collaboration required by two players to finish a given game level. We first asked a game level designer to create playable game level chunks. Then, two artificial ...
Comments
Information & Contributors
Information
Published In
Copyright © 2022 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 04 May 2022
Published in PACMCGIT Volume 5, Issue 1
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 66Total Downloads
- Downloads (Last 12 months)16
- Downloads (Last 6 weeks)0
Reflects downloads up to 09 Aug 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in