EFfECT-RL: Enabling Framework for Establishing Causality and Triggering engagement through RL
Pages 4836 - 4843
Abstract
Skill-based games offer an exceptional avenue for entertainment, fostering self-esteem, relaxation, and social satisfaction. Engagement in online skill gaming platforms is however heavily dependent on the outcomes and experience (e.g., wins/losses). Understanding the factors driving increased engagement is crucial within skill gaming platforms. In this study, we aim to address two key questions: (1) "What factors are driving users to increase their engagement?" and (2) "How can we personalize users journey accordingly to further optimize their engagement?". In skill gaming platforms, the impact of causal relationships often manifests with a delay, which varies significantly as users? personas evolve. Without a detailed information on treatments (such as timing and frequency), estimating the impact of a causal-treatment-effect in a highly volatile game-play data becomes exceedingly challenging. This work proposes a framework called EFfECT-RL that establishes causal discovery by integrating change-point detection and explainable K-means clustering, while leveraging users' game-play and transactional-data. Unlike existing methods which were unable to detect causal-effects in extremely volatile-data, EFfECT-RL generates threshold-trees (~ 79% accuracy) elucidating causal-relationships. Once the causal relationship is established, we personalize treatments by developing a novel offline deep reinforcement learning-based approach. Our online recommendations show a 3% improvement in user engagement (platform-centric) with 70% relevancy (user-centric).
References
[1]
M Mehdi Afsar, Trafford Crump, and Behrouz Far. 2022. Reinforcement learning based recommender systems: A survey. Comput. Surveys 55, 7 (2022), 1--38.
[2]
Saeed Aghabozorgi, Ali Seyed Shirkhorshidi, and Teh Ying Wah. 2015. Time-series clustering--a decade review. Information systems 53 (2015), 16--38.
[3]
Chainarong Amornbunchornvej, Elena Zheleva, and Tanya Y Berger-Wolf. 2019. Variable-lag granger causality for time series analysis. In 2019 IEEE International Conference on Data Science and Advanced Analytics (DSAA). IEEE, 21--30.
[4]
Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. 2017. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine 34, 6 (2017), 26--38.
[5]
Surajit Chakrabarty, Rukma Talwadker, and Tridib Mukherjee. 2021. ScarceGAN: Discriminative Classification Framework for Rare Class Identification for Longitudinal Data with Weak Prior. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 140--150.
[6]
Arthur Chatton, Florent Le Borgne, Clémence Leyrat, Florence Gillaizeau, Chloé Rousseau, Laetitia Barbin, David Laplaud, Maxime Léger, Bruno Giraudeau, and Yohann Foucher. 2020. G-computation, propensity score-based methods, and targeted maximum likelihood estimator for causal inference with different covariates sets: a comparative simulation study. Scientific reports 10, 1 (2020), 9219.
[7]
Linda M Collins, Susan A Murphy, and Victor Strecher. 2007. The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): new methods for more potent eHealth interventions. American journal of preventive medicine 32, 5 (2007), S112--S118.
[8]
Rajeev H Dehejia and Sadek Wahba. 2002. Propensity score-matching methods for nonexperimental causal studies. Review of Economics and statistics 84, 1 (2002), 151--161.
[9]
Jianqing Fan, Zhaoran Wang, Yuchen Xie, and Zhuoran Yang. 2020. A theoretical analysis of deep Q-learning. In Learning for Dynamics and Control. PMLR, 486--489.
[10]
Scott Fitzgerald. 2019. Over-the-top video services in India: Media imperialism after globalization. Media Industries Journal 6, 2 (2019), 00--00.
[11]
Chris Fregly and Antje Barth. 2021. Data Science on AWS. " O'Reilly Media, Inc.".
[12]
Scott Fujimoto, Herke Hoof, and David Meger. 2018. Addressing function approximation error in actor-critic methods. In International conference on machine learning. PMLR, 1587--1596.
[13]
Rong Gao, Haifeng Xia, Jing Li, Donghua Liu, Shuai Chen, and Gang Chun. 2019. DRCGR: Deep reinforcement learning framework incorporating CNN and GAN-based for interactive recommendation. In 2019 IEEE International Conference on Data Mining (ICDM). IEEE, 1048--1053.
[14]
Ivo Grondman, Lucian Busoniu, Gabriel AD Lopes, and Robert Babuska. 2012. A survey of actor-critic reinforcement learning: Standard and natural policy gradients. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 42, 6 (2012), 1291--1307.
[15]
Minghao Han, Lixian Zhang, Jun Wang, and Wei Pan. 2020. Actor-critic reinforcement learning for control with stability guarantee. IEEE Robotics and Automation Letters 5, 4 (2020), 6217--6224.
[16]
Jason S Haukoos and Roger J Lewis. 2015. The propensity score. Jama 314, 15 (2015), 1637--1638.
[17]
Miguel A Hernán, Babette Brumback, and James M Robins. 2001. Marginal structural models to estimate the joint causal effect of nonrandomized treatments. J. Amer. Statist. Assoc. 96, 454 (2001), 440--448.
[18]
Daniel N Hill, Robert Moakler, Alan E Hubbard, Vadim Tsemekhman, Foster Provost, and Kiril Tsemekhman. 2015. Measuring causal impact of online actions via natural experiments: Application to display advertising. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1839--1847.
[19]
Bibhudutta Jena. 2019. An Approach for Forecast Prediction in Data Analytics Field by Tableau Software. International Journal of Information Engineering & Electronic Business 11, 1 (2019).
[20]
Rebecca Killick, Paul Fearnhead, and Idris A Eckley. 2012. Optimal detection of change points with a linear computational cost. J. Amer. Statist. Assoc. 107, 500 (2012), 1590--1598.
[21]
Tatiana Kossaczká, Matthias Ehrhardt, and Michael Günther. 2023. Deep FDM: Enhanced finite difference methods by deep learning. Franklin Open 4 (2023), 100039.
[22]
Randall J LeVeque. 1998. Finite difference methods for differential equations. Draft version for use in AMath 585, 6 (1998), 112.
[23]
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. 2020. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 (2020).
[24]
Fan Li, Kari Lock Morgan, and Alan M Zaslavsky. 2018. Balancing covariates via propensity score weighting. J. Amer. Statist. Assoc. 113, 521 (2018), 390--400.
[25]
Giampiero Marra and Rosalba Radice. 2011. A flexible instrumental variable approach. Statistical Modelling 11, 6 (2011), 581--603.
[26]
Cristian Martín, Peter Langendoerfer, Pouya Soltani Zarrin, Manuel Díaz, and Bartolomé Rubio. 2022. Kafka-ML: Connecting the data stream with ML/AI frameworks. Future Generation Computer Systems 126 (2022), 15--33.
[27]
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013).
[28]
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015. Human-level control through deep reinforcement learning. nature 518, 7540 (2015), 529--533.
[29]
Michal Moshkovitz, Sanjoy Dasgupta, Cyrus Rashtchian, and Nave Frost. 2020. Explainable k-means and k-medians clustering. In International conference on machine learning. PMLR, 7055--7065.
[30]
Koyel Mukherjee, Deepanshi Seth, Prachi Mittal, Nuthi S Gowtham, Tridib Mukherjee, Dattatreya Biswas, and Sanjay Agrawal. 2022. ComParE: A User Behavior Centric Framework for Personalized Recommendations in Skill Gaming Platforms. In 5th Joint International Conference on Data Science & Management of Data (9th ACM IKDD CODS and 27th COMAD). 186--194.
[31]
Abiodun E Onile, Ram Machlev, Eduard Petlenkov, Yoash Levron, and Juri Belikov. 2021. Uses of the digital twins concept for energy services, intelligent recommendation systems, and demand side management: A review. Energy Reports 7 (2021), 997--1015.
[32]
Stephen O'Neill, Noémi Kreif, Richard Grieve, Matthew Sutton, and Jasjeet S Sekhon. 2016. Estimating causal effects: considering three alternatives to difference-in-differences estimation. Health Services and Outcomes Research Methodology 16 (2016), 1--21.
[33]
Justus J Randolph and Kristina Falbe. 2014. A step-by-step guide to propensity score matching in R. Practical Assessment, Research & Evaluation 19 (2014).
[34]
Debanjan Sadhukhan, Sachin Kumar, Swarit Sankule, and Tridib Mukherjee. 2023. t-RELOAD: A REinforcement Learning-based Recommendation for Outcome-driven Application. In Proceedings of the Third International Conference on AI-ML Systems. 1--7.
[35]
Yonadav Shavit, Benjamin Edelman, and Brian Axelrod. 2020. Causal strategic linear regression. In International Conference on Machine Learning. PMLR, 8676--8686.
[36]
Ali Shojaie and Emily B Fox. 2022. Granger causality: A review and recent advances. Annual Review of Statistics and Its Application 9 (2022), 289--319.
[37]
Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press.
[38]
Rukma Talwadker, Surajit Chakrabarty, Aditya Pareek, Tridib Mukherjee, and Deepak Saini. 2022. Cognitionnet: A collaborative neural network for play style discovery in online skill gaming platform. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 3961--3969.
[39]
Zhiqiang Tan. 2006. Regression and weighting methods for causal inference using instrumental variables. J. Amer. Statist. Assoc. 101, 476 (2006), 1607--1618.
[40]
Sam O Tickle, IA Eckley, Paul Fearnhead, and Kaylea Haynes. 2020. Parallelization of a common changepoint detection method. Journal of Computational and Graphical Statistics 29, 1 (2020), 149--161.
[41]
Hado Van Hasselt, Arthur Guez, and David Silver. 2016. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, Vol. 30.
[42]
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. 2016. Dueling network architectures for deep reinforcement learning. In International conference on machine learning. PMLR, 1995--2003.
[43]
Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning 8, 3 (1992), 229--256.
[44]
Liuyi Yao, Zhixuan Chu, Sheng Li, Yaliang Li, Jing Gao, and Aidong Zhang. 2021. A survey on causal inference. ACM Transactions on Knowledge Discovery from Data (TKDD) 15, 5 (2021), 1--46.
[45]
Yisong Yue and Thorsten Joachims. 2009. Interactively optimizing information retrieval systems as a dueling bandits problem. In Proceedings of the 26th Annual International Conference on Machine Learning. 1201--1208.
[46]
Alessio Zanga, Elif Ozkirimli, and Fabio Stella. 2022. A survey on causal discovery: Theory and practice. International Journal of Approximate Reasoning 151 (2022), 101--129.
[47]
Xiangyu Zhao, Liang Zhang, Zhuoye Ding, Long Xia, Jiliang Tang, and Dawei Yin. 2018. Recommendations with negative feedback via pairwise deep reinforcement learning. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1040--1048.
[48]
Jichen Zhu and Santiago Ontañón. 2020. Player-centered AI for automatic game personalization: Open problems. In International Conference on the Foundations of Digital Games. 1--8.
[49]
Seyedjamal Zolhavarieh, Saeed Aghabozorgi, Ying Wah Teh, et al . 2014. A review of subsequence time series clustering. The Scientific World Journal 2014 (2014).
[50]
Lixin Zou, Long Xia, Zhuoye Ding, Jiaxing Song, Weidong Liu, and Dawei Yin. 2019. Reinforcement learning to optimize long-term user engagement in recommender systems. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. 2810--2818.
Index Terms
- EFfECT-RL: Enabling Framework for Establishing Causality and Triggering engagement through RL
Recommendations
Expertise and engagement: re-designing citizen science games with players' minds in mind
FDG '19: Proceedings of the 14th International Conference on the Foundations of Digital GamesMany studies have already shown that games can be a useful tool to make boring or difficult tasks more engaging. However, with serious game design being a relatively nascent field, such experiences can still be hard to learn and not very motivating. In ...
Towards Accelerating Offline RL based Recommender Systems
AIMLSystems '21: Proceedings of the First International Conference on AI-ML SystemsReinforcement learning optimizes an objective function by learning an optimal policy for taking a sequence of actions in an environment. Offline RL is a mechanism to learn policy offline from pre-generated traces of an agent’s interactions in an ...
A unified framework for recommendations based on quaternary semantic analysis
SIGIR '11: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information RetrievalSocial network systems such as FaceBook and YouTube have played a significant role in capturing both explicit and implicit user preferences for different items in the form of ratings and tags. This forms a quaternary relationship among users, items, ...
Comments
Information & Contributors
Information
Published In
October 2024
5705 pages
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 21 October 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
CIKM '24
Sponsor:
CIKM '24: The 33rd ACM International Conference on Information and Knowledge Management
October 21 - 25, 2024
ID, Boise, USA
Acceptance Rates
Overall Acceptance Rate 1,861 of 8,427 submissions, 22%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 52Total Downloads
- Downloads (Last 12 months)52
- Downloads (Last 6 weeks)3
Reflects downloads up to 08 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in