short-paper

On Modeling Long-Term User Engagement from Stochastic Feedback

Authors:

Guoxi Zhang,

Xing Yao,

Xuanji XiaoAuthors Info & Claims

WWW '23 Companion: Companion Proceedings of the ACM Web Conference 2023

Pages 940 - 944

https://doi.org/10.1145/3543873.3587626

Published: 30 April 2023 Publication History

Get Access

Abstract

An ultimate goal of recommender systems (RS) is to improve user engagement. Reinforcement learning (RL) is a promising paradigm for this goal, as it directly optimizes overall performance of sequential recommendation. However, many existing RL-based approaches induce huge computational overhead, because they require not only the recommended items but also all other candidate items to be stored. This paper proposes an efficient alternative that does not require the candidate items. The idea is to model the correlation between user engagement and items directly from data. Moreover, the proposed approach consider randomness in user feedback and termination behavior, which are ubiquitous for RS but rarely discussed in RL-based prior work. With online A/B experiments on real-world RS, we confirm the efficacy of the proposed approach and the importance of modeling the two types of randomness.

References

[1]

Marc G. Bellemare, Will Dabney, and Rémi Munos. 2017. A Distributional Perspective on Reinforcement Learning. In Proceedings of the Thirty-fourth International Conference on Machine Learning (Sydney, Australia). PMLR, 449–458.

Google Scholar

[2]

Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed H. Chi. 2019. Top-K Off-Policy Correction for a REINFORCE Recommender System. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining (Melbourne, VIC, Australia). Association for Computing Machinery, New York, NY, USA, 456–464.

Digital Library

Google Scholar

[3]

Will Dabney, Mark Rowland, Marc G. Bellemare, and Rémi Munos. 2018. Distributional Reinforcement Learning With Quantile Regression. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (New Orleans, LA, USA). 2892–2901.

Crossref

Google Scholar

[4]

Eugene Ie, Vihan Jain, Jing Wang, Sanmit Narvekar, Ritesh Agarwal, Rui Wu, Heng-Tze Cheng, Tushar Chandra, and Craig Boutilier. 2019. SlateQ: A Tractable Decomposition for Reinforcement Learning with Recommendation Sets. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence. International Joint Conferences on Artificial Intelligence Organization, 2592–2599.

Crossref

Google Scholar

[5]

Michael L. Littman. 1994. Markov Games As a Framework for Multi-agent Reinforcement Learning. In Proceedings of the Eleventh International Conference on International Conference on Machine Learning (New Brunswick, NJ, USA). 157–163.

Digital Library

Google Scholar

[6]

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. 2015. Human-level control through deep reinforcement learning. Nature 518, 7540 (2015), 529–533.

Google Scholar

[7]

Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G. Azzolini, Dmytro Dzhulgakov, Andrey Mallevich, Ilia Cherniavskii, Yinghai Lu, Raghuraman Krishnamoorthi, Ansha Yu, Volodymyr Kondratenko, Stephanie Pereira, Xianjie Chen, Wenlin Chen, Vijay Rao, Bill Jia, Liang Xiong, and Misha Smelyanskiy. 2019. Deep Learning Recommendation Model for Personalization and Recommendation Systems. arxiv:1906.00091 [cs.IR]

Google Scholar

[8]

Kai Wang, Zhene Zou, Yue Shang, Qilin Deng, Minghao Zhao, Runze Wu, Xudong Shen, Tangjie Lyu, and Changjie Fan. 2021. RL4RS: A Real-World Benchmark for Reinforcement Learning based Recommender System. arxiv:2110.11073 [cs.IR]

Google Scholar

[9]

Martha White. 2017. Unifying Task Specification in Reinforcement Learning. In Proceedings of the Thirty-Fourth International Conference on Machine Learning (Sydney, Australia). PMLR, 3742–3750.

Google Scholar

[10]

Zhongwen Xu, Hado P van Hasselt, and David Silver. 2018. Meta-Gradient Reinforcement Learning. In Advances in Neural Information Processing Systems. Curran Associates, Inc.

Google Scholar

[11]

Naoto Yoshida, Eiji Uchibe, and Kenji Doya. 2013. Reinforcement learning with state-dependent discount factor. In 2013 IEEE third joint international conference on development and learning and epigenetic robotics. IEEE, 1–6.

Google Scholar

[12]

Xiangyu Zhao, Changsheng Gu, Haoshenglun Zhang, Xiaobing Liu, Xiwang Yang, and Jiliang Tang. 2019. Deep Reinforcement Learning for Online Advertising in Recommender Systems. arXiv preprint arXiv:1909.03602 (2019).

Google Scholar

[13]

Xiangyu Zhao, Long Xia, Liang Zhang, Zhuoye Ding, Dawei Yin, and Jiliang Tang. 2018. Deep Reinforcement Learning for Page-Wise Recommendations. In Proceedings of the 12th ACM Conference on Recommender Systems (Vancouver, BC, Canada). Association for Computing Machinery, 95–103.

Digital Library

Google Scholar

[14]

Guanjie Zheng, Fuzheng Zhang, Zihan Zheng, Yang Xiang, Nicholas Jing Yuan, Xing Xie, and Zhenhui Li. 2018. DRN: A Deep Reinforcement Learning Framework for News Recommendation. In Proceedings of the 2018 World Wide Web Conference (Lyon, France). International World Wide Web Conferences Steering Committee, 167–176.

Digital Library

Google Scholar

Cited By

View all

Ding RXie RHao XYang XGe KZhang XZhou JLin L(2023)Interpretable User Retention Modeling in RecommendationProceedings of the 17th ACM Conference on Recommender Systems10.1145/3604915.3608818(702-708)Online publication date: 14-Sep-2023
https://dl.acm.org/doi/10.1145/3604915.3608818

Index Terms

On Modeling Long-Term User Engagement from Stochastic Feedback
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Markov decision processes
2. Information systems
  1. World Wide Web
    1. Web searching and information discovery
      1. Content ranking
      2. Personalization

Recommendations

Investigating serendipity in recommender systems based on real user feedback
SAC '18: Proceedings of the 33rd Annual ACM Symposium on Applied Computing

Over the past several years, research in recommender systems has emphasized the importance of serendipity, but there is still no consensus on the definition of this concept and whether serendipitous items should be recommended is still not a well-...
Predicting User Engagement in Twitter with Collaborative Ranking
RecSysChallenge '14: Proceedings of the 2014 Recommender Systems Challenge

Collaborative Filtering (CF) is a core component of popular web-based services such as Amazon, YouTube, Netflix, and Twitter. Most applications use CF to recommend a small set of items to the user. For instance, YouTube presents to a user a list of top-...
Reinforcement learning for addressing the cold-user problem in recommender systems
Abstract
Recommender systems are widely used in webshops because of their ability to provide users with personalized recommendations. However, the cold-user problem (i.e., recommending items to new users) is an important issue many webshops face. With the ...

Comments

Information & Contributors

Information

Published In

WWW '23 Companion: Companion Proceedings of the ACM Web Conference 2023

April 2023

1567 pages

ISBN:9781450394192

DOI:10.1145/3543873

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 April 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper
Research
Refereed limited

Conference

WWW '23

Sponsor:

SIGWEB

WWW '23: The ACM Web Conference 2023

April 30 - May 4, 2023

TX, Austin, USA

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
45
Total Downloads

Downloads (Last 12 months)23
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Ding RXie RHao XYang XGe KZhang XZhou JLin L(2023)Interpretable User Retention Modeling in RecommendationProceedings of the 17th ACM Conference on Recommender Systems10.1145/3604915.3608818(702-708)Online publication date: 14-Sep-2023
https://dl.acm.org/doi/10.1145/3604915.3608818

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Abstract

References

Cited By

Index Terms

Recommendations

Investigating serendipity in recommender systems based on real user feedback

Predicting User Engagement in Twitter with Collaborative Ranking

Reinforcement learning for addressing the cold-user problem in recommender systems

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Get Access

Login options

Full Access

View options

PDF

eReader

HTML Format

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations