EvalRS 2023: Well-Rounded Recommender Systems for Real-World Deployments
Authors: 
Federico Bianchi, Patrick John Chia, Jacopo Tagliabue, Ciro Greco, Gabriel S P Moreira, Davide Eynard, Fahd Husain, 
Claudio PomoAuthors Info & Claims
Federico Bianchi, Patrick John Chia, Jacopo Tagliabue, Ciro Greco, Gabriel S P Moreira, Davide Eynard, Fahd Husain, Claudio PomoAuthors Info & ClaimsPages 5851 - 5852
Abstract
EvalRS aims to bring together practitioners from industry and academia to foster a debate on rounded evaluation of recommender systems, with a focus on real-world impact across a multitude of deployment scenarios. Recommender systems are often evaluated only through accuracy metrics, which fall short of fully characterizing their generalization capabilities and miss important aspects, such as fairness, bias, usefulness, informativeness. This workshop builds on the success of last year's workshop at CIKM, but with a broader scope and an interactive format.
References
[1]
Vito Walter Anelli, Alejandro Bellog'i n, Tommaso Di Noia, Dietmar Jannach, and Claudio Pomo. 2022. Top-N Recommendation Algorithms: A Quest for the State-of-the-Art. In UMAP. ACM, 121--131.
[2]
Federico Bianchi, Jacopo Tagliabue, and Bingqing Yu. 2021. Query2Prod2Vec: Grounded Word Embeddings for eCommerce. In NAACL-HLT (Industry Papers). Association for Computational Linguistics, 154--162.
[3]
Patrick John Chia, Jacopo Tagliabue, Federico Bianchi, Chloe He, and Brian Ko. 2022. Beyond NDCG: Behavioral Testing of Recommender Systems with RecList. In WWW (Companion Volume). ACM, 99--104.
[4]
Editors. 2023. Algorithmic recommendations, anyone? Nature Machine Intelligence (2023). https://doi.org/10.1038/s42256-023-00631--7
[5]
Karl Higley, Even Oldridge, Ronay Ak, Sara Rabhi, and Gabriel de Souza Pereira Moreira. 2022. Building and Deploying a Multi-Stage Recommender System with Merlin. In RecSys. ACM, 632--635.
[6]
Dietmar Jannach, Gabriel de Souza Pereira Moreira, and Even Oldridge. 2020. Why Are Deep Learning Models Not Consistently Winning Recommender Systems Competitions Yet?: A Position Paper. In RecSys Challenge. ACM, 44--49.
[7]
Dietmar Jannach, Pearl Pu, Francesco Ricci, and Markus Zanker. 2021. Recommender Systems: Past, Present, Future. AI Mag., Vol. 42, 3 (2021), 3--6.
[8]
Pigi Kouki, Ilias Fountalis, Nikolaos Vasiloglou, Xiquan Cui, Edo Liberty, and Khalifeh Al Jadda. 2020. From the lab to production: A case study of session-based recommendations in the home-improvement domain. In RecSys. ACM, 140--149.
[9]
Ahmed Rashed, Shayan Jawed, Lars Schmidt-Thieme, and Andre Hintsches. 2020. MultiRec: A Multi-Relational Approach for Unique Item Recommendation in Auction Systems. In RecSys. ACM, 230--239.
[10]
Zhu Sun, Di Yu, Hui Fang, Jie Yang, Xinghua Qu, Jie Zhang, and Cong Geng. 2020. Are We Evaluating Rigorously? Benchmarking Recommendation for Reproducible Evaluation and Fair Comparison. In RecSys. ACM, 23--32.
[11]
Jacopo Tagliabue. 2021. You Do Not Need a Bigger Boat: Recommendations at Reasonable Scale in a (Mostly) Serverless and Open Stack. In RecSys. ACM, 598--600.
[12]
Jacopo Tagliabue, Federico Bianchi, Tobias Schnabel, Giuseppe Attanasio, Ciro Greco, Gabriel de Souza Moreira, and Patrick John Chia. 2023. A challenge for rounded evaluation of recommender systems. Nature Machine Intelligence (2023), 1--2.
[13]
Jacopo Tagliabue, Federico Bianchi, Tobias Schnabel, Giuseppe Attanasio, Ciro Greco, Gabriel de Souza P. Moreira, and Patrick John Chia. 2022. EvalRS: a rounded evaluation of recommender systems. In CIKM Workshops (CEUR Workshop Proceedings, Vol. 3318). CEUR-WS.org.
Index Terms
- EvalRS 2023: Well-Rounded Recommender Systems for Real-World Deployments
Recommendations
Investigating serendipity in recommender systems based on real user feedback
SAC '18: Proceedings of the 33rd Annual ACM Symposium on Applied ComputingOver the past several years, research in recommender systems has emphasized the importance of serendipity, but there is still no consensus on the definition of this concept and whether serendipitous items should be recommended is still not a well-...
Third Workshop: Perspectives on the Evaluation of Recommender Systems (PERSPECTIVES 2023)
RecSys '23: Proceedings of the 17th ACM Conference on Recommender SystemsEvaluation is important when developing and deploying recommender systems. The PERSPECTIVES workshop sheds light on the different, potentially diverging or contradictory perspectives on the evaluation of recommender systems. Building on the discussions ...
Comments
Information & Contributors
Information
Published In
August 2023
5996 pages
ISBN:9798400701030
DOI:10.1145/3580305
- General Chairs:
- Ambuj Singh,
- Yizhou Sun,
- Program Chairs:
- Leman Akoglu,
- Dimitrios Gunopulos,
- Xifeng Yan,
- Ravi Kumar,
- Fatma Ozcan,
- Jieping Ye
Copyright © 2023 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 04 August 2023
Check for updates
Author Tags
Qualifiers
- Abstract
Funding Sources
- Hoffman?Yee Research Grants Program and the Stanford Institute for Human-Centered Artificial Intelligence
Conference
KDD '23: The 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
August 6 - 10, 2023
CA, Long Beach, USA
Acceptance Rates
Overall Acceptance Rate 1,133 of 8,635 submissions, 13%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 128Total Downloads
- Downloads (Last 12 months)53
- Downloads (Last 6 weeks)2
Reflects downloads up to 09 Jan 2025
Other Metrics
Citations
Cited By
View allView Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in