research-article

Public Access

Player Modeling via Multi-Armed Bandits

Authors:

Robert C. Gray,

Danielle Arigo,

Santiago OntañónAuthors Info & Claims

FDG '20: Proceedings of the 15th International Conference on the Foundations of Digital Games

Article No.: 7, Pages 1 - 8

https://doi.org/10.1145/3402942.3402952

Published: 17 September 2020 Publication History

All formats PDF

Abstract

This paper focuses on building personalized player models solely from player behavior in the context of adaptive games. We present two main contributions: The first is a novel approach to player modeling based on multi-armed bandits (MABs). This approach addresses, at the same time and in a principled way, both the problem of collecting data to model the characteristics of interest for the current player and the problem of adapting the interactive experience based on this model. Second, we present an approach to evaluating and fine-tuning these algorithms prior to generating data in a user study. This is an important problem, because conducting user studies is an expensive and labor-intensive process; therefore, an ability to evaluate the algorithms beforehand can save a significant amount of resources. We evaluate our approach in the context of modeling players’ social comparison orientation (SCO) and present empirical results from both simulations and real players.

References

[1]

Justin T Alexander, John Sear, and Andreas Oikonomou. 2013. An investigation of the effects of game difficulty on player enjoyment. Entertainment computing 4, 1 (2013), 53–62.

[2]

P. Auer, N. Cesa-Bianchi, and P. Fischer. 2002. Finite-time analysis of the multiarmed bandit problem. Machine learning 47, 2 (2002), 235–256.

[3]

Joseph Bates. 1992. Virtual reality, art, and entertainment. Presence: Teleoperators & Virtual Environments 1, 1(1992), 133–138.

[4]

Abraham P. Buunk and Frederick X. Gibbons. 2007. Social comparison: The end of a theory and the emergence of a field. Organ. Behav. Hum. Decis. Process. 102, 1 (2007), 3–21.

[5]

Karina Caro, Yuanyuan Feng, Timothy Day, Evan Freed, Boyd Fox, and Jichen Zhu. 2018. Understanding the effect of existing positive relationships on a social motion-based game for health. In Proceedings of the 12th EAI International Conference on Pervasive Computing Technologies for Healthcare. 77–87.

Digital Library

[6]

Ben Carterette, Evangelos Kanoulas, and Emine Yilmaz. 2011. Simulating simple user behavior for system effectiveness evaluation. In Proceedings of the 20th ACM international conference on Information and knowledge management. 611–620.

Digital Library

[7]

Zhengxing Chen, Su Xue, John Kolen, Navid Aghdaie, Kazi A Zaman, Yizhou Sun, and Magy Seif El-Nasr. 2017. Eomm: An engagement optimized matchmaking framework. In Proceedings of the 26th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 1143–1150.

Digital Library

[8]

Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. 2011. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics. 208–214.

[9]

Anders Drachen, Alessandro Canossa, and Georgios N Yannakakis. 2009. Player modeling using self-organization in Tomb Raider: Underworld. In 2009 IEEE symposium on computational intelligence and games. IEEE, 1–8.

[10]

Evan M Forman, Stephanie G Kerrigan, Meghan L Butryn, Adrienne S Juarascio, Stephanie M Manasse, Santiago Ontañón, Diane H Dallal, Rebecca J Crochiere, and Danielle Moskow. 2019. Can the artificial intelligence technique of reinforcement learning use continuously-monitored digital data to optimize treatment for weight loss?Journal of behavioral medicine 42, 2 (2019), 276–290.

[11]

Brinton J. Keating M. Furberg, R. and A. Ortiz. 2016. Crowd-sourced Fitbit datasets 03.12.2016-05.12.2016. http://doi.org/10.5281/zenodo.53894

[12]

Krzysztof Z Gajos and Krysta Chauncey. 2017. The influence of personality traits and cognitive load on the use of adaptive user interfaces. In Proceedings of the 22nd International Conference on Intelligent User Interfaces. 301–306.

Digital Library

[13]

Frederick X. Gibbons and Bram P. Buunk. 1999. Individual differences in social comparison: Development of a scale of social comparison orientation. J. Pers. Soc. Psychol. 76, 1 (1999), 129–142.

[14]

Robert C Gray. 2018. Adaptive Game Input Using Knowledge of Player Capability: Designing for Individuals with Different Abilities. MS Thesis. Drexel University.

[15]

Robert C Gray, Jichen Zhu, and Santiago Ontañón. 2020. Regression Oracles and Exploration Strategies for Short-Horizon Multi-Armed Bandits. Proceedings of the 2020 IEEE Conference on Games (CoG) (2020), accepted.

[16]

J-L Guo, C Fan, and Z-H Guo. 2011. Weblog patterns and human dynamics with decreasing interest. The European Physical Journal B 81, 3 (2011), 341.

[17]

C. Holmgård, A. Liapis, J. Togelius, and G. N. Yannakakis. 2014. Evolving personas for player decision modeling. In 2014 IEEE Conference on Computational Intelligence and Games. 1–8.

[18]

William M Klein. 1997. Objective standards are not enough: affective, self-evaluative, and behavioral responses to social comparison information.Journal of personality and social psychology 72, 4(1997), 763.

[19]

Michael Mateas and Andrew Stern. 2003. Façade: An experiment in building a fully-realized interactive drama. In Game developers conference, Vol. 2. 4–8.

[20]

Santiago Ontañón. 2017. Combinatorial multi-armed bandits for real-time strategy games. Journal of Artificial Intelligence Research 58 (2017), 665–702.

Digital Library

[21]

Michael S Orendurff, Jason A Schoen, Greta C Bernatz, Ava D Segal, and Glenn K Klute. 2008. How humans walk: bout duration, steps per bout, and rest duration.Journal of Rehabilitation Research & Development 45, 7(2008).

[22]

Mark O Riedl, Andrew Stern, Don Dini, and Jason Alderman. 2008. Dynamic experience management in virtual worlds for entertainment, education, and training. International Transactions on Systems Science and Applications, Special Issue on Agent Based Systems for Human Learning 4, 2 (2008), 23–42.

[23]

Manu Sharma, Santiago Ontañón, Manish Mehta, and Ashwin Ram. 2010. Drama management and player modeling for interactive fiction games. Computational Intelligence 26, 2 (2010), 183–211.

[24]

David Thue and Vadim Bulitko. 2018. Toward a Unified Understanding of Experience Management. In Fourteenth Artificial Intelligence and Interactive Digital Entertainment Conference.

[25]

Josep Valls-Vargas, Andrew Kahl, Justin Patterson, Glen Muschio, Aroutis Foster, and Jichen Zhu. 2015. Designing and tracking play styles in solving the incognitum. In Proceedings of the Games+ Learning+ Society Conference, to appear. Games+ Learning+ Society.

[26]

Josep Valls-Vargas, Santiago Ontanón, and Jichen Zhu. 2015. Exploring player trace segmentation for dynamic play style prediction. In Eleventh Artificial Intelligence and Interactive Digital Entertainment Conference.

[27]

Viktor Wendel, Johannes Alef, Stefan Göbel, and Ralf Steinemtz. 2014. A method for simulating players in a collaborative multiplayer serious game. In Proceedings of the 2014 ACM International Workshop on Serious Games. 15–20.

Digital Library

[28]

Peter William Weyhrauch. 1997. Guiding interactive drama. Carnegie Mellon University.

Digital Library

[29]

Joanne V Wood. 1989. Theory and research concerning social comparisons of personal attributes.Psychol. Bull. 106, 2 (1989), 231.

[30]

Georgios N Yannakakis and John Hallam. 2004. Evolving opponents for interesting interactive computer games. From animals to animats 8 (2004), 499–508.

[31]

Georgios N Yannakakis, Pieter Spronck, Daniele Loiacono, and Elisabeth André. 2013. Player modeling. (2013).

[32]

J Zhang, D Brackbill, S Yang, J Becker, N Herbert, and D Centola. 2016. Support or competition? How online social networks increase physical activity: a randomized controlled trial. Prev. Med. reports 4(2016), 453–458.

[33]

Jichen Zhu, Yuanyuan Feng, Anushay Furqan, Robert C Gray, Timothy Day, Jessica Nebolsky, and Karina Caro. 2018. Towards Extending Social Exergame Engagement with Agents. In Companion of the 2018 ACM Conference on Computer Supported Cooperative Work and Social Computing. 349–352.

[34]

Jichen Zhu and Santiago Ontañón. 2019. Experience management in multi-player games. In 2019 IEEE Conference on Games (CoG). IEEE, 1–6.

Digital Library

Cited By

Arigo DGray RDallal DVillareale JZhu J(2023)Selection of and Response to Physical Activity–Based Social Comparisons in a Digital Environment: Series of Daily Assessment StudiesJMIR Human Factors10.2196/4123910(e41239)Online publication date: 27-Feb-2023
https://doi.org/10.2196/41239
Vandenberg JMin WCatete VBoulden DMott B(2023)Leveraging Game Design Activities for Middle Grades AI Education in Rural CommunitiesProceedings of the 18th International Conference on the Foundations of Digital Games10.1145/3582437.3587193(1-4)Online publication date: 12-Apr-2023
https://dl.acm.org/doi/10.1145/3582437.3587193
Villareale JMaram SSeif El-Nasr MZhu J(2023)Integrating Players’ Perspectives in AI-Based Games: Case Studies of Player-AI Interaction DesignProceedings of the 18th International Conference on the Foundations of Digital Games10.1145/3582437.3582451(1-9)Online publication date: 12-Apr-2023
https://dl.acm.org/doi/10.1145/3582437.3582451
Show More Cited By

Recommendations

Multiplayer Modeling via Multi-Armed Bandits
2021 IEEE Conference on Games (CoG)
This paper focuses on player modeling in multiplayer adaptive games. While player modeling has received a significant amount of attention, less is known about how to use player modeling in multiplayer games, especially when an experience management AI ...
Multi-armed bandits with episode context

A multi-armed bandit episode consists of n trials, each allowing selection of one of K arms, resulting in payoff from a distribution over [0,1] associated with that arm. We assume contextual side information is available at the start of the episode. ...
Player modeling using self-organization in tomb raider: underworld
CIG'09: Proceedings of the 5th international conference on Computational Intelligence and Games

We present a study focused on constructing models of players for the major commercial title Tomb Raider: Underworld (TRU). Emergent self-organizing maps are trained on high-level playing behavior data obtained from 1365 players that completed the TRU ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

FDG '20: Proceedings of the 15th International Conference on the Foundations of Digital Games

September 2020

804 pages

ISBN:9781450388078

DOI:10.1145/3402942

Editors:
Georgios N. Yannakakis
University of Malta
,
Antonios Liapis
University of Malta
,
Penny Kyburz
Australian National University
,
Vanessa Volz
Modl.AI
,
Foaad Khosmood
California Polytechnic State University
,
Phil Lopes
École Polytechnique Fédèrale de Lausanne

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 September 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

National Science Foundation

Conference

FDG '20

FDG '20: International Conference on the Foundations of Digital Games

September 15 - 18, 2020

Bugibba, Malta

Acceptance Rates

Overall Acceptance Rate 152 of 415 submissions, 37%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
512
Total Downloads

Downloads (Last 12 months)213
Downloads (Last 6 weeks)26

Reflects downloads up to 12 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Arigo DGray RDallal DVillareale JZhu J(2023)Selection of and Response to Physical Activity–Based Social Comparisons in a Digital Environment: Series of Daily Assessment StudiesJMIR Human Factors10.2196/4123910(e41239)Online publication date: 27-Feb-2023
https://doi.org/10.2196/41239
Vandenberg JMin WCatete VBoulden DMott B(2023)Leveraging Game Design Activities for Middle Grades AI Education in Rural CommunitiesProceedings of the 18th International Conference on the Foundations of Digital Games10.1145/3582437.3587193(1-4)Online publication date: 12-Apr-2023
https://dl.acm.org/doi/10.1145/3582437.3587193
Villareale JMaram SSeif El-Nasr MZhu J(2023)Integrating Players’ Perspectives in AI-Based Games: Case Studies of Player-AI Interaction DesignProceedings of the 18th International Conference on the Foundations of Digital Games10.1145/3582437.3582451(1-9)Online publication date: 12-Apr-2023
https://dl.acm.org/doi/10.1145/3582437.3582451
Gray RVillareale JFox TDallal DOntanon SArigo DJabbari SZhu J(2023)Improving Fairness in Adaptive Social Exergames via Shapley BanditsProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584050(322-336)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3581641.3584050
Gray RZhu JOntañón S(2023)Beyond UCT: MAB Exploration Improvements for Monte Carlo Tree Search2023 IEEE Conference on Games (CoG)10.1109/CoG57401.2023.10333233(1-8)Online publication date: 21-Aug-2023
https://doi.org/10.1109/CoG57401.2023.10333233
Kantharaju PAlderfer KZhu JChar BSmith BOntanon S(2022)Modeling Player Knowledge in a Parallel Programming Educational GameIEEE Transactions on Games10.1109/TG.2020.303750514:1(64-75)Online publication date: Mar-2022
https://doi.org/10.1109/TG.2020.3037505
de Lima ESilva BGalam G(2022)Adaptive virtual reality horror games based on Machine learning and player modelingEntertainment Computing10.1016/j.entcom.2022.10051543(100515)Online publication date: Aug-2022
https://doi.org/10.1016/j.entcom.2022.100515
Zhu JDallal DGray RVillareale JOntañón SForman EArigo D(2021)Personalization Paradox in Behavior Change AppsProceedings of the ACM on Human-Computer Interaction10.1145/34491905:CSCW1(1-21)Online publication date: 22-Apr-2021
https://dl.acm.org/doi/10.1145/3449190
Ontanon SZhu JVerbert KParra DHammond TKnijnenburg BO'Donovan JTaele P(2021)The Personalization Paradox: the Conflict between Accurate User Models and Personalized Adaptive SystemsCompanion Proceedings of the 26th International Conference on Intelligent User Interfaces10.1145/3397482.3450734(64-66)Online publication date: 14-Apr-2021
https://dl.acm.org/doi/10.1145/3397482.3450734
Yang ZOntañón S(2021)Contextual Combinatorial Bandits in Real-Time Strategy Games2021 IEEE Conference on Games (CoG)10.1109/CoG52621.2021.9619063(1-9)Online publication date: 17-Aug-2021
https://dl.acm.org/doi/10.1109/CoG52621.2021.9619063
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents