Minimizing Live Experiments in Recommender Systems: User Simulation to Evaluate Preference Elicitation Policies
Authors: 
Chih-Wei Hsu, Martin Mladenov, Ofer Meshi, James Pine, Hubert Pham, Shane Li, Xujian Liang, Anton Polishko, Li Yang, Ben Scheetz, 
Craig BoutilierAuthors Info & Claims
Chih-Wei Hsu, Martin Mladenov, Ofer Meshi, James Pine, Hubert Pham, Shane Li, Xujian Liang, Anton Polishko, Li Yang, Ben Scheetz, Craig BoutilierAuthors Info & ClaimsPages 2925 - 2929
Abstract
Evaluation of policies in recommender systems typically involves A/B live experiments on real users to assess a new policy's impact on relevant metrics. This "gold standard'' comes at a high cost, however, in terms of cycle time, user cost, and potential user retention. In developing policies for onboarding users, these costs can be especially problematic, since on-boarding occurs only once. In this work, we describe a simulation methodology used to augment (and reduce) the use of live experiments. We illustrate its deployment for the evaluation of preference elicitation algorithms used to onboard new users of the YouTube Music platform. By developing counterfactually robust user behavior models, and a simulation service that couples such models with production infrastructure, we can test new algorithms in a way that reliably predicts their performance on key metrics when deployed live.
References
[1]
John Ahlgren, Kinga Bojarczuk, Sophia Drossopoulou, Inna Dvortsova, Johann George, Natalija Gucevska, Mark Harman, Maria Lomeli, Simon Mark Lucas, Erik Meijer, Steve Omohundro, Rubmary Rojas, Silvia Sapora, Jie M. Zhang, and Norm Zhou. 2021. Facebook's Cyber-Cyber and Cyber-Physical Digital Twins. In Proceedings of the International Conference on Evaluation and Assessment in Software Engineering (EASE-21). 1--9.
[2]
Craig Boutilier. 2002. A POMDP Formulation of Preference Elicitation Problems. In Proceedings of the Eighteenth National Conference on Artificial Intelligence (AAAI-02). Edmonton, 239--246.
[3]
Urszula Chajewska, Daphne Koller, and Ronald Parr. 2000. Making Rational Decisions Using Adaptive Utility Elicitation. In Proceedings of the Seventeenth National Conference on Artificial Intelligence (AAAI-00). Austin, TX, 363--369.
[4]
Bassam H. Chaptini. 2005. Use of Discrete Choice Models with Recommender Systems. Ph.,D. Dissertation. Massachusetts Institute of Technology, Cambridge, MA.
[5]
Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed Chi. 2018. Top-K Off-Policy Correction for a REINFORCE Recommender System. In 12th ACM International Conference on Web Search and Data Mining (WSDM-19). Melbourne, Australia, 456--464.
[6]
Miroslav Dudík, John Langford, and Lihong Li. 2011. Doubly Robust Policy Evaluation and Learning. In Proceedings of the Twenty-eighth International Conference on Machine Learning (ICML-11). Bellevue, WA, 1097--1104.
[7]
Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2016. Session-based Recommendations with Recurrent Neural Networks. In Proceedings of the Fourth International Conference on Learning Representations (ICLR-16). San Juan, Puerto Rico.
[8]
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation, Vol. 9, 8 (1997), 1735--1780.
[9]
J. J. Hopfield. 1982. Neural Networks and Physical Systems with Emergent Collective Computational Abilities. Proceedings of the National Academy of Sciences, Vol. 79, 8 (1982), 2554--2558.
[10]
Eugene Ie, Vihan Jain, Jing Wang, Sanmit Narvekar, Ritesh Agarwal, Rui Wu, Heng-Tze Cheng, Tushar Chandra, and Craig Boutilier. 2019a. SlateQ: A Tractable Decomposition for Reinforcement Learning with Recommendation Sets. In Proceedings of the Twenty-eighth International Joint Conference on Artificial Intelligence (IJCAI-19). Macau, 2592--2599.
[11]
Eugene Ie, Chih wei Hsu, Martin Mladenov, Vihan Jain, Sanmit Narvekar, Jing Wang, Rui Wu, and Craig Boutilier. 2019b. RecSim: A Configurable Simulation Platform for Recommender Systems. (2019). arXiv:1909.04847.
[12]
Q. Liu, S. Wu, D. Wang, Z. Li, and L. Wang. 2016. Context-Aware Sequential Recommendation. In Proceedings of the IEEE International Conference on Data Mining (ICDM-16). Barcelona, 1053--1058.
[13]
Jordan J. Louviere, David A. Hensher, and Joffre D. Swait. 2000. Stated Choice Methods: Analysis and Application. Cambridge University Press, Cambridge.
[14]
Sean M. McNee, Shyong K. Lam, Joseph A. Konstan, and John Riedl. 2003. Interfaces for Eliciting New User Preferences in Recommender Systems. In Proceedings of the 9th International Conference on User Modeling (UM-03). Johnstown, PA, 178--187.
[15]
Ofer Meshi, Jon Feldman, Li Yang, Ben Scheetz, Yanli Cai, Mohammadhossein Bateni, Corbyn Salisbury, Vikram Aggarwal, and Craig Boutilier. 2023. Preference Elicitation for Music Recommendations. In ICML 2023 Workshop The Many Facets of Preference-Based Learning.
[16]
Martin Mladenov, Elliot Creager, Kevin Swerksy, Omer Ben-Porat, Richard S. Zemel, and Craig Boutilier. 2020. Optimizing Long-term Social Welfare in Recommender Systems: A Constrained Matching Approach. In Proceedings of the Thirty-seventh International Conference on Machine Learning (ICML-20). Vienna, 6987--6998.
[17]
Martin Mladenov, Chih-wei Hsu, Vihan Jain, Eugene Ie, Chris Colby, Nic Mayoraz, Hubert Pham, Dustin Tran, Ivan Vendrov, and Craig Boutilier. 2021. RecSim NG: Toward Principled Uncertainty Modeling for Recommender Ecosystems. (2021). arXiv:2103.08057.
[18]
Massimo Quadrana, Alexandros Karatzoglou, Balázs Hidasi, and Paolo Cremonesi. 2017. Personalizing Session-based Recommendations with Hierarchical Recurrent Neural Networks. In Proceedings of the Eleventh ACM Conference on Recommender Systems (RecSys-17). 130--137.
[19]
David Rohde, Stephen Bonner, Travis Dunlop, Flavian Vasile, and Alexandros Karatzoglou. 2018. RecoGym: A Reinforcement Learning Environment for the problem of Product Recommendation in Online Advertising. (2018). arXiv:1808.00720 [cs.IR].
[20]
Guy Shani and Asela Gunawardana. 2011. Evaluating Recommendation Systems. In Recommender Systems Handbook, F. Ricci, L. Rokach, B. Shapira, and P. Kantor (Eds.). Springer, Boston, 257--297.
[21]
Paolo Viappiani and Craig Boutilier. 2010. Optimal Bayesian Recommendation Sets and Myopically Optimal Choice Query Sets. In Advances in Neural Information Processing Systems 23 (NIPS). Vancouver, 2352--2360.
[22]
Yueqi Wang, Yoni Halpern, Shuo Chang, Jingchen Feng, Elaine Ya Le, Longfei Li, Xujian Liang, Min-Cheng Huang, Shane Li, Alex Beutel, Yaping Zhang, and Shuchao Bi. 2023. Learning from Negative User Feedback and Measuring Responsiveness for Sequential Recommenders. In Proceedings of the 17th ACM Conference on Recommender Systems (RecSys-23). Singapore, 1049--1053.
[23]
Siriu Yao, Yoni Halpern, Nithum Thain, Xuezhi Wang, Kang Lee, Flavien Prost, Ed H. Chi, Jilin Chen, and Alex Beutel. 2020. Measuring Recommender System Effects with Simulated Users. In 2nd Workshop on Fairness, Accountability, Transparency, Ethics and Society on the Web (FATES-20). Taipei.
[24]
Fajie Yuan, Alexandros Karatzoglou, Ioannis Arapakis, Joemon M. Jose, and Xiangnan He. 2019. A Simple Convolutional Generative Network for Next Item Recommendation. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining (WSDM-19). Melbourne, 582--590.
[25]
Kesen Zhao, Shuchang Liu, Qingpeng Cai, Xiangyu Zhao, Ziru Liu, Dong Zheng, Peng Jiang, and Kun Gai. 2023. KuaiSim: A Comprehensive Simulator for Recommender Systems. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023. New Orleans.
[26]
Xiangyu Zhao, Long Xia, Liang Zhang, Zhuoye Ding, Dawei Yin, and Jiliang Tang. 2018. Deep Reinforcement Learning for Page-wise Recommendations. In Proceedings of the 12th ACM Conference on Recommender Systems (RecSys-18). Vancouver, 95--103.
Recommendations
Semi-supervised discriminative preference elicitation for cold-start recommendation
CIKM '13: Proceedings of the 22nd ACM international conference on Information & Knowledge ManagementRecommendation for cold users is fairly challenging because no prior rating can be used in preference prediction. To tackle this cold-start scenario, rating elicitation is usually employed through an initial interview in which users are queried by some ...
Preference elicitation for narrowing the recommended list for groups
RecSys '14: Proceedings of the 8th ACM Conference on Recommender systemsA group may appreciate recommendations on items that fit their joint preferences. When the members' actual preferences are unknown, a recommendation can be made with the aid of collaborative filtering methods. We offer to narrow down the recommended ...
Using Groups of Items for Preference Elicitation in Recommender Systems
CSCW '15: Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social ComputingTo achieve high quality initial personalization, recommender systems must provide an efficient and effective process for new users to express their preferences. We propose that this goal is best served not by the classical method where users begin by ...
Comments
Information & Contributors
Information
Published In
July 2024
3164 pages
ISBN:9798400704314
DOI:10.1145/3626772
- General Chairs:
- Grace Hui Yang,
- Hongning Wang,
- Sam Han,
- Program Chairs:
- Claudia Hauff,
- Guido Zuccon,
- Yi Zhang
Copyright © 2024 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 July 2024
Check for updates
Author Tags
Qualifiers
- Short-paper
Conference
SIGIR 2024
Sponsor:
SIGIR 2024: The 47th International ACM SIGIR Conference on Research and Development in Information Retrieval
July 14 - 18, 2024
Washington DC, USA
Acceptance Rates
Overall Acceptance Rate 792 of 3,983 submissions, 20%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 175Total Downloads
- Downloads (Last 12 months)175
- Downloads (Last 6 weeks)81
Reflects downloads up to 03 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in