short-paper

User-centric evaluation of session-based recommendations for an automated radio station

Authors:

Dietmar JannachAuthors Info & Claims

RecSys '19: Proceedings of the 13th ACM Conference on Recommender Systems

Pages 516 - 520

https://doi.org/10.1145/3298689.3347046

Published: 10 September 2019 Publication History

Abstract

The creation of an automated and virtually endless playlist given a start item is a common feature of modern media streaming services. When no past information about the user's preferences is available, the creation of such playlists can be done using session-based recommendation techniques. In this case, the recommendations only depend on the start item and the user's interactions in the current listening session, such as "liking" or skipping an item.

In recent years, various novel session-based techniques were proposed, often based on deep learning. The evaluation of such approaches is in most cases solely based on offline experimentation and abstract accuracy measures. However, such evaluations cannot inform us about the quality as perceived by users. To close this research gap, we have conducted a user study (N=250), where the participants interacted with an automated online radio station. Each treatment group received recommendations that were generated by one of five different algorithms. Our results show that comparably simple techniques led to quality perceptions that are similar or even better than when a complex deep learning mechanism or Spotify's recommendations are used. The simple mechanisms, however, often tend to recommend comparably popular tracks, which can lead to lower discovery effects.

Supplementary Material

ZIP File (p516-ludewig.zip)

The auxiliary material contains a PDF document with an additional result table that supplements the findings reported in the paper.

Download
388.91 KB

References

[1]

Luke Barrington, Reid Oda, and Gert R. G. Lanckriet. 2009. Smarter than Genius? Human Evaluation of Music Recommender Systems. In Proc. ISMIR '09. 357--362.

[2]

Jöran Beel and Stefan Langer. 2015. A Comparison of Offline Evaluations, Online Evaluations, and User Studies in the Context of Research-Paper Recommender Systems. In Proc. TPDL '15.

[3]

Geoffray Bonnin and Dietmar Jannach. 2014. Automated Generation of Music Playlists: Survey and Experiments. Comput. Surveys 47, 2 (2014), 26:1--26:35.

Digital Library

[4]

Shuo Chen, Josh L. Moore, Douglas Turnbull, and Thorsten Joachims. 2012. Playlist Prediction via Metric Embedding. In Proc. KDD '12. 714--722.

Digital Library

[5]

Paolo Cremonesi, Franca Garzotto, and Roberto Turrin. 2012. Investigating the Persuasion Potential of Recommender Systems from a Quality Perspective: An Empirical Study. Transactions on Interactive Intelligent Systems 2, 2, Article 11 (June 2012), 11:1--11:41 pages.

Digital Library

[6]

Michael D. Ekstrand, F. Maxwell Harper, Martijn C. Willemsen, and Joseph A. Konstan. 2014. User Perception of Differences in Recommender Algorithms. In Proc. RecSys '14. 161--168.

Digital Library

[7]

Florent Garcin, Boi Faltings, Olivier Donatsch, Ayar Alazzawi, Christophe Bruttin, and Amr Huber. 2014. Offline and Online Evaluation of News Recommender Systems at Swissinfo.ch. In Proc. RecSys '14.

Digital Library

[8]

Negar Hariri, Bamshad Mobasher, and Robin Burke. 2014. Context Adaptation in Interactive Recommender Systems. In Proc. RecSys '14. 41--48.

Digital Library

[9]

Balázs Hidasi and Alexandros Karatzoglou. 2017. Recurrent Neural Networks with Top-k Gains for Session-based Recommendations. CoRR abs/1706.03847 (2017).

[10]

Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2016. Session-based Recommendations with Recurrent Neural Networks. In Proc. ICLR '16.

[11]

Dietmar Jannach, Lukas Lerche, and Michael Jugovac. 2015. Item Familiarity as a Possible Confounding Factor in User-Centric Recommender Systems Evaluation. i-com Journal of Interactive Media 14, 1 (2015), 29--39.

[12]

Iman Kamehkhosh, Geoffray Bonnin, and Dietmar Jannach. 2019. Effects of Recommendations on the Playlist Creation Behavior of Users. User Modeling and User-Adapted Interaction May (2019).

[13]

Iman Kamehkhosh and Dietmar Jannach. 2017. User Perception of Next-Track Music Recommendations. In Proc. UMAP '17.

Digital Library

[14]

Bart P. Knijnenburg, Martijn C. Willemsen, Zeno Gantner, Hakan Soncu, and Chris Newell. 2012. Explaining the user experience of recommender systems. User Modeling and User-Adapted Interaction 22, 4 (2012), 441--504.

Digital Library

[15]

Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tao Lian, and Jun Ma. 2017. Neural Attentive Session-based Recommendation. In Proc. CIKM '17. 1419--1428.

Digital Library

[16]

Qiao Liu, Yifu Zeng, Refuoe Mokhosi, and Haibin Zhang. 2018. STAMP: Short-Term Attention/Memory Priority Model for Session-based Recommendation. In Proc. KDD '18. 1831--1839.

Digital Library

[17]

Benedikt Loepp, Tim Donkers, Timm Kleemann, and Jürgen Ziegler. 2018. Impact of Item Consumption on Assessment of Recommendations in User Studies. In Proc. RecSys '18. 49--53.

Digital Library

[18]

Malte Ludewig and Dietmar Jannach. 2018. Evaluation of Session-based Recommendation Algorithms. User-Modeling and User-Adapted Interaction 28, 4--5 (2018), 331--390.

Digital Library

[19]

Malte Ludewig, Iman Kamehkhosh, Nick Landia, and Dietmar Jannach. 2018. Effective Nearest-Neighbor Music Recommendations. In Proc. RecSys '18 Challenge Workshop at ACM RecSys.

Digital Library

[20]

Andrii Maksai, Florent Garcin, and Boi Faltings. 2015. Predicting Online Performance of News Recommender Systems Through Richer Evaluation Metrics. In Proc. RecSys '15.

Digital Library

[21]

Bamshad Mobasher, Honghua Dai, Tao Luo, and Miki Nakagawa. 2002. Using Sequential and Non-Sequential Patterns in Predictive Web Usage Mining Tasks. In Proc. ICDM '02. 669--672.

Digital Library

[22]

Pearl Pu, Li Chen, and Rong Hu. 2011. A User-centric Evaluation Framework for Recommender Systems. In Proc. RecSys '11. 157--164.

Digital Library

[23]

Massimo Quadrana, Paolo Cremonesi, and Dietmar Jannach. 2018. Sequence-Aware Recommender Systems. Comput. Surveys 51 (2018), 1--36. Issue 4.

Digital Library

[24]

Marco Rossetti, Fabio Stella, and Markus Zanker. 2016. Contrasting Offline and Online Results when Evaluating Recommendation Algorithms. In Proc. RecSys '16.

Digital Library

[25]

Guy Shani, David Heckerman, and Ronen I. Brafman. 2005. An MDP-Based Recommender System. Journal of Machine Learning Researh 6 (2005), 1265--1295.

Digital Library

[26]

Longqi Yang, Michael Sobolev, Christina Tsangouri, and Deborah Estrin. 2018. Understanding User Interactions with Podcast Recommendations Delivered via Voice. In Proc. RecSys '18. 190--194.

Digital Library

[27]

Yuan Yao and F. Maxwell Harper. 2018. Judging Similarity: A User-centric Study of Related Item Recommendations. In Proc. RecSys '18. 288--296.

Digital Library

[28]

Yuan Yao and F. Maxwell Harper. 2018. Judging Similarity: A User-centric Study of Related Item Recommendations. In Proc. RecSys '18. 288--296.

Digital Library

[29]

Qian Zhao, Martijn C. Willemsen, Gediminas Adomavicius, F. Maxwell Harper, and Joseph A. Konstan. 2018. Interpreting User Inaction in Recommender Systems. In Proc. RecSys '18. 40--48.

Digital Library

Cited By

Kelen DBenczúr A(2022)A probabilistic perspective on nearest neighbor for implicit recommendationInternational Journal of Data Science and Analytics10.1007/s41060-022-00367-416:2(217-235)Online publication date: 29-Oct-2022
https://doi.org/10.1007/s41060-022-00367-4
Kouki PFountalis IVasiloglou NCui XLiberty EAl Jadda K(2020)From the lab to production: A case study of session-based recommendations in the home-improvement domainFourteenth ACM Conference on Recommender Systems10.1145/3383313.3412235(140-149)Online publication date: 22-Sep-2020
https://dl.acm.org/doi/10.1145/3383313.3412235
Sharakhina LMikhailov NSelyankina KSemkina A(2020)Digital Technologies in Development of Modern Music Industry2020 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (EIConRus)10.1109/EIConRus49466.2020.9039328(71-76)Online publication date: Jan-2020
https://doi.org/10.1109/EIConRus49466.2020.9039328
Show More Cited By

Index Terms

User-centric evaluation of session-based recommendations for an automated radio station
1. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Recommender systems
    2. Specialized information retrieval
      1. Multimedia and multimodal retrieval
        Music retrieval
  2. World Wide Web
    1. Web searching and information discovery
      1. Collaborative filtering

Recommendations

News Session-Based Recommendations using Deep Neural Networks
DLRS 2018: Proceedings of the 3rd Workshop on Deep Learning for Recommender Systems

News recommender systems are aimed to personalize users experiences and help them to discover relevant articles from a large and dynamic search space. Therefore, news domain is a challenging scenario for recommendations, due to its sparse user profiling, ...
Long-tail Session-based Recommendation
RecSys '20: Proceedings of the 14th ACM Conference on Recommender Systems

Session-based recommendation focuses on the prediction of user actions based on anonymous sessions and is a necessary method in the lack of user historical data. However, none of the existing session-based recommendation methods explicitly takes the ...
Combining User-Based and Session-Based Recommendations with Recurrent Neural Networks
Neural Information Processing
Abstract
Recommender systems generate recommendations based on user profiles, which consist of past interactions of users with items. When user profiles are not available, session-based recommendation can be used instead to make predictions based on ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

RecSys '19: Proceedings of the 13th ACM Conference on Recommender Systems

September 2019

635 pages

ISBN:9781450362436

DOI:10.1145/3298689

General Chairs:
Toine Bogers
Aalborg University Copenhagen, Denmark
,
Alan Said
University of Gothenburg, Sweden
,
Program Chairs:
Peter Brusilovsky
University of Pittsburgh
,
Domonkos Tikk
Gravity R&D, Hungary

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 September 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

RecSys '19

RecSys '19: Thirteenth ACM Conference on Recommender Systems

September 16 - 20, 2019

Copenhagen, Denmark

Acceptance Rates

RecSys '19 Paper Acceptance Rate 36 of 189 submissions, 19%;

Overall Acceptance Rate 254 of 1,295 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
476
Total Downloads

Downloads (Last 12 months)21
Downloads (Last 6 weeks)1

Reflects downloads up to 12 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kelen DBenczúr A(2022)A probabilistic perspective on nearest neighbor for implicit recommendationInternational Journal of Data Science and Analytics10.1007/s41060-022-00367-416:2(217-235)Online publication date: 29-Oct-2022
https://doi.org/10.1007/s41060-022-00367-4
Kouki PFountalis IVasiloglou NCui XLiberty EAl Jadda K(2020)From the lab to production: A case study of session-based recommendations in the home-improvement domainFourteenth ACM Conference on Recommender Systems10.1145/3383313.3412235(140-149)Online publication date: 22-Sep-2020
https://dl.acm.org/doi/10.1145/3383313.3412235
Sharakhina LMikhailov NSelyankina KSemkina A(2020)Digital Technologies in Development of Modern Music Industry2020 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (EIConRus)10.1109/EIConRus49466.2020.9039328(71-76)Online publication date: Jan-2020
https://doi.org/10.1109/EIConRus49466.2020.9039328
Ludewig MMauro NLatifi SJannach D(2020)Empirical analysis of session-based recommendation algorithmsUser Modeling and User-Adapted Interaction10.1007/s11257-020-09277-131:1(149-181)Online publication date: 20-Oct-2020
https://dl.acm.org/doi/10.1007/s11257-020-09277-1
Jannach DMobasher BBerkovsky S(2020)Research directions in session-based and sequential recommendationUser Modeling and User-Adapted Interaction10.1007/s11257-020-09274-430:4(609-616)Online publication date: 6-Aug-2020
https://doi.org/10.1007/s11257-020-09274-4
Jannach DJugovac M(2019)Measuring the Business Value of Recommender SystemsACM Transactions on Management Information Systems10.1145/337008210:4(1-23)Online publication date: 10-Dec-2019
https://dl.acm.org/doi/10.1145/3370082
Jannach DQuadrana MCremonesi P(2012)Session-Based Recommender SystemsRecommender Systems Handbook10.1007/978-1-0716-2197-4_8(301-334)Online publication date: 24-Feb-2012
https://doi.org/10.1007/978-1-0716-2197-4_8

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents