Cited By
View all- Dinnissen K(2024)Fairness and Transparency in Music Recommender Systems: Improvements for ArtistsProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688024(1368-1375)Online publication date: 8-Oct-2024
Putting Popularity Bias Mitigation to the Test: A User-Centric Evaluation in Music Recommenders
Authors: 
Robin Ungruh, 
Karlijn Dinnissen, Anja Volk, 
Maria Soledad Pera, Hanna HauptmannAuthors Info & Claims
Robin Ungruh, Karlijn Dinnissen, Anja Volk, Maria Soledad Pera, Hanna HauptmannAuthors Info & ClaimsPages 169 - 178
Abstract
Popularity bias is a prominent phenomenon in recommender systems (RS), especially in the music domain. Although popularity bias mitigation techniques are known to enhance the fairness of RS while maintaining their high performance, there is a lack of understanding regarding users’ actual perception of the suggested music. To address this gap, we conducted a user study (n=40) exploring user satisfaction and perception of personalized music recommendations generated by algorithms that explicitly mitigate popularity bias. Specifically, we investigate item-centered and user-centered bias mitigation techniques, aiming to ensure fairness for artists or users, respectively. Results show that neither mitigation technique harms the users’ satisfaction with the recommendation lists despite promoting underrepresented items. However, the item-centered mitigation technique impacts user perception; by promoting less popular items, it reduces users’ familiarity with the items. Lower familiarity evokes discovery—the feeling that the recommendations enrich the user’s taste. We demonstrate that this can ultimately lead to higher satisfaction, highlighting the potential of less-popular recommendations to improve the user experience.
Supplemental Material
PDF File
Additional information about the statistical tests and metrics computed on the results from the user study.
- Download
- 470.00 KB
References
[1]
Himan Abdollahpouri. 2020. Popularity bias in recommendation: A multi-stakeholder perspective. Ph. D. Dissertation. University of Colorado at Boulder.
[2]
Himan Abdollahpouri and Robin Burke. 2021. Multistakeholder recommender systems. In Recommender systems handbook. Springer, 647–677.
[3]
Himan Abdollahpouri, Robin Burke, and Bamshad Mobasher. 2017. Controlling popularity bias in learning-to-rank recommendation. In Proceedings of the eleventh ACM conference on recommender systems. 42–46.
[4]
Himan Abdollahpouri, Robin Burke, and Bamshad Mobasher. 2019. Managing popularity bias in recommender systems with personalized re-ranking. arXiv preprint arXiv:1901.07555 (2019).
[5]
Himan Abdollahpouri, Masoud Mansoury, Robin Burke, and Bamshad Mobasher. 2019. The unfairness of popularity bias in recommendation. arXiv preprint arXiv:1907.13286 (2019).
[6]
Himan Abdollahpouri, Masoud Mansoury, Robin Burke, and Bamshad Mobasher. 2020. The connection between popularity bias, calibration, and fairness in recommendation. In Proceedings of the 14th ACM Conference on Recommender Systems.
[7]
Himan Abdollahpouri, Masoud Mansoury, Robin Burke, Bamshad Mobasher, and Edward Malthouse. 2021. User-centered evaluation of popularity bias in recommender systems. In Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization. 119–129.
[8]
Dmitry Bogdanov, Martín Haro, Ferdinand Fuhrmann, Anna Xambó, Emilia Gómez, and Perfecto Herrera. 2013. Semantic audio content-based music recommendation and visualization based on user preference examples. Information Processing & Management 49, 1 (2013), 13–33.
[9]
CBS Interactive. 2024. Last.fm. https://www.last.fm/
[10]
Oscar Celma. 2010. Music recommendation and discovery: The long tail, long fail, and long play in the digital music space. Springer Science & Business Media.
[11]
Òscar Celma and Pedro Cano. 2008. From hits to niches? or how popular artists can bias music recommendation and discovery. In Proceedings of the 2nd KDD Workshop on Large-Scale Recommender Systems and the Netflix Prize Competition.
[12]
Savvina Daniil, Mirjam Cuper, Cynthia CS Liem, Jacco van Ossenbruggen, and Laura Hollink. 2024. Reproducing popularity bias in recommendation: The effect of evaluation strategies. ACM Transactions on Recommender Systems 2, 1 (2024), 1–39.
[13]
Yashar Deldjoo, Dietmar Jannach, Alejandro Bellogin, Alessandro Diffonzo, and Dario Zanzonelli. 2022. Fairness in Recommender Systems: Research Landscape and Future Directions. (2022).
[14]
Karlijn Dinnissen and Christine Bauer. 2023. Amplifying artists’ voices: Item provider perspectives on influence and fairness of music streaming platforms. In Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization. 238–249.
[15]
Michael D Ekstrand, Anubrata Das, Robin Burke, Fernando Diaz, 2022. Fairness in information access systems. Foundations and Trends® in Information Retrieval 16, 1-2 (2022), 1–177.
[16]
Bruce Ferwerda, Mark P Graus, Andreu Vall, Marko Tkalcic, and Markus Schedl. 2017. How item discovery enabled by diversity leads to increased recommendation list attractiveness. In Proceedings of the Symposium on Applied Computing.
[17]
Bruce Ferwerda, Eveline Ingesson, Michaela Berndl, and Markus Schedl. 2023. I Don’t Care How Popular You Are! Investigating Popularity Bias in Music Recommendations from a User’s Perspective. In Proceedings of the 2023 Conference on Human Information Interaction and Retrieval (Austin, TX, USA) (CHIIR ’23). 357–361.
[18]
Andrew V Goldberg. 1997. An efficient implementation of a scaling minimum-cost flow algorithm. Journal of algorithms 22, 1 (1997), 1–29.
[19]
Mark P Graus and Bruce Ferwerda. 2021. The Moderating Effect of Active Engagement on Appreciation of Popularity in Song Recommendations. In Diversity, Divergence, Dialogue: 16th International Conference, iConference 2021, Beijing, China, March 17–31, 2021, Proceedings, Part I 16. Springer, 364–374.
[20]
Christian Hansen, Rishabh Mehrotra, Casper Hansen, Brian Brost, Lucas Maystre, and Mounia Lalmas. 2021. Shifting consumption towards diverse content on music streaming platforms. In Proceedings of the 14th ACM international conference on web search and data mining. 238–246.
[21]
F Maxwell Harper and Joseph A Konstan. 2015. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis) 5, 4 (2015), 1–19.
[22]
Li-tze Hu and Peter M Bentler. 1999. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural equation modeling: a multidisciplinary journal 6, 1 (1999), 1–55.
[23]
IFPI. 2023. IFPI Global Music Report 2022. https://globalmusicreport.ifpi.org/
[24]
Dietmar Jannach, Lukas Lerche, Iman Kamehkhosh, and Michael Jugovac. 2015. What recommenders recommend: an analysis of recommendation biases and possible countermeasures. User Modeling and User-Adapted Interaction 25 (2015).
[25]
Yitong Ji, Aixin Sun, Jie Zhang, and Chenliang Li. 2020. A re-visit of the popularity baseline in recommender systems. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 1749–1752.
[26]
Iman Kamehkhosh and Dietmar Jannach. 2017. User perception of next-track music recommendations. In Proceedings of the 25th conference on user modeling, adaptation and personalization. 113–121.
[27]
Anastasiia Klimashevskaia, Mehdi Elahi, Dietmar Jannach, Lars Skjærven, Astrid Tessem, and Christoph Trattner. 2023. Evaluating The Effects of Calibrated Popularity Bias Mitigation: A Field Study. In Proceedings of the 17th ACM Conference on Recommender Systems. 1084–1089.
[28]
Anastasiia Klimashevskaia, Mehdi Elahi, Dietmar Jannach, Christoph Trattner, and Lars Skjærven. 2022. Mitigating Popularity Bias in Recommendation: Potential and Limits of Calibration Approaches. In Advances in Bias and Fairness in Information Retrieval: Third International Workshop, BIAS 2022, Stavanger, Norway, April 10, 2022, Revised Selected Papers. Springer, 82–90.
[29]
Bart P Knijnenburg, Martijn C Willemsen, Zeno Gantner, Hakan Soncu, and Chris Newell. 2012. Explaining the user experience of recommender systems. User modeling and user-adapted interaction 22 (2012), 441–504.
[30]
Yehuda Koren, Steffen Rendle, and Robert Bell. 2021. Advances in collaborative filtering. Recommender systems handbook (2021), 91–142.
[31]
Dominik Kowald, Gregor Mayr, Markus Schedl, and Elisabeth Lex. 2023. A Study on Accuracy, Miscalibration, and Popularity Bias in Recommendations. arXiv preprint arXiv:2303.00400 (2023).
[32]
Dominik Kowald, Markus Schedl, and Elisabeth Lex. 2020. The unfairness of popularity bias in music recommendation: A reproducibility study. In Advances in Information Retrieval: 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14–17, 2020, Proceedings, Part II 42. Springer, 35–42.
[33]
Karl Krauth, Sarah Dean, Alex Zhao, Wenshuo Guo, Mihaela Curmei, Benjamin Recht, and Michael I Jordan. 2020. Do offline metrics predict online performance in recommender systems?arXiv preprint arXiv:2011.07931 (2020).
[34]
Oleg Lesota, Stefan Brandl, Matthias Wenzel, Alessandro B Melchiorre, Elisabeth Lex, Navid Rekabsaz, and Markus Schedl. 2022. Exploring Cross-group Discrepancies in Calibrated Popularity for Accuracy/Fairness Trade-off Optimization. In CEUR Workshop Proceedings, Vol. 3268. RWTH Aachen.
[35]
Oleg Lesota, Gustavo Escobedo, Yashar Deldjoo, Bruce Ferwerda, Simone Kopeinik, Elisabeth Lex, Navid Rekabsaz, and Markus Schedl. 2023. Computational Versus Perceived Popularity Miscalibration in Recommender Systems. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1889–1893.
[36]
Oleg Lesota, Alessandro Melchiorre, Navid Rekabsaz, Stefan Brandl, Dominik Kowald, Elisabeth Lex, and Markus Schedl. 2021. Analyzing item popularity bias of music recommender systems: are different genders equally affected?. In Proceedings of the 15th ACM Conference on Recommender Systems. 601–606.
[37]
Yu Liang and Martijn C Willemsen. 2022. Exploring the longitudinal effects of nudging on users’ music genre exploration behavior and listening preferences. In Proceedings of the 16th ACM Conference on Recommender Systems. 3–13.
[38]
Alessandro B Melchiorre, Navid Rekabsaz, Emilia Parada-Cabaleiro, Stefan Brandl, Oleg Lesota, and Markus Schedl. 2021. Investigating gender fairness of recommendation algorithms in the music domain. Information Processing & Management 58, 5 (2021), 102666.
[39]
Daniel Müllensiefen, Bruno Gingras, Jason Musil, and Lauren Stewart. 2014. The musicality of non-musicians: An index for assessing musical sophistication in the general population. PloS one 9, 2 (2014), e89642.
[40]
Yoon-Joo Park and Alexander Tuzhilin. 2008. The long tail of recommender systems and how to leverage it. In Proceedings of the 2008 ACM conference on Recommender systems. 11–18.
[41]
Lorenzo Porcaro, Emilia Gómez, and Carlos Castillo. 2024. Assessing the impact of music recommendation diversity on listeners: A longitudinal study. ACM Transactions on Recommender Systems 2, 1 (2024), 1–47.
[42]
Pearl Pu, Li Chen, and Rong Hu. 2011. A user-centric evaluation framework for recommender systems. In Proceedings of the fifth ACM conference on Recommender systems. 157–164.
[43]
Francesco Ricci, Lior Rokach, and Bracha Shapira. 2021. Recommender systems: Techniques, applications, and challenges. Recommender Systems Handbook (2021), 1–35.
[44]
Rebecca Salganik, Fernando Diaz, and Golnoosh Farnadi. 2023. Fairness Through Domain Awareness: Mitigating Popularity Bias For Music Discovery. arXiv preprint arXiv:2308.14601 (2023).
[45]
Markus Schedl, Stefan Brandl, Oleg Lesota, Emilia Parada-Cabaleiro, David Penz, and Navid Rekabsaz. 2022. LFM-2B: a dataset of enriched music listening events for recommender systems research and fairness analysis. In ACM SIGIR Conference on Human Information Interaction and Retrieval. 337–341.
[46]
Markus Schedl, Peter Knees, Brian McFee, and Dmitry Bogdanov. 2021. Music recommendation systems: Techniques, use cases, and challenges. In Recommender Systems Handbook. Springer, 927–971.
[47]
Markus Schedl, Hamed Zamani, Ching-Wei Chen, Yashar Deldjoo, and Mehdi Elahi. 2018. Current challenges and visions in music recommender systems research. International Journal of Multimedia Information Retrieval 7 (2018), 95–116.
[48]
Spotify AB. 2024. Spotify for Developers. https://developer.spotify.com/
[49]
Harald Steck. 2018. Calibrated recommendations. In Proceedings of the 12th ACM conference on recommender systems. 154–162.
[50]
Gábor Takács and Domonkos Tikk. 2012. Alternating least squares for personalized ranking. In Proceedings of the sixth ACM conference on Recommender systems. 83–90.
[51]
Emre Yalcin and Alper Bilge. 2022. Evaluating unfairness of popularity bias in recommender systems: A comprehensive user-centric analysis. Information Processing & Management 59, 6 (2022), 103100.
[52]
Meike Zehlike, Francesco Bonchi, Carlos Castillo, Sara Hajian, Mohamed Megahed, and Ricardo Baeza-Yates. 2017. Fa* ir: A fair top-k ranking algorithm. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. 1569–1578.
[53]
Yang Zhang, Fuli Feng, Xiangnan He, Tianxin Wei, Chonggang Song, Guohui Ling, and Yongdong Zhang. 2021. Causal intervention for leveraging popularity bias in recommendation. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 11–20.
[54]
Xiangyu Zhao, Zhendong Niu, and Wei Chen. 2013. Opinion-based collaborative filtering to solve popularity bias in recommender systems. In Database and Expert Systems Applications: 24th International Conference, DEXA 2013, Prague, Czech Republic, August 26-29, 2013. Proceedings, Part II 24. Springer, 426–433.
Index Terms
- Putting Popularity Bias Mitigation to the Test: A User-Centric Evaluation in Music Recommenders
Recommendations
I Don’t Care How Popular You Are! Investigating Popularity Bias in Music Recommendations from a User’s Perspective
CHIIR '23: Proceedings of the 2023 Conference on Human Information Interaction and RetrievalRecommender systems are designed to help us navigate through an abundance of online content. Collaborative filtering (CF) approaches are commonly used to leverage behaviors of others with a similar taste to make predictions for the target user. However, ...
The Unfairness of Popularity Bias in Music Recommendation: A Reproducibility Study
Advances in Information RetrievalAbstractResearch has shown that recommender systems are typically biased towards popular items, which leads to less popular items being underrepresented in recommendations. The recent work of Abdollahpouri et al. in the context of movie recommendations ...
User-centered Evaluation of Popularity Bias in Recommender Systems
UMAP '21: Proceedings of the 29th ACM Conference on User Modeling, Adaptation and PersonalizationRecommendation and ranking systems are known to suffer from popularity bias; the tendency of the algorithm to favor a few popular items while under-representing the majority of other items. Prior research has examined various approaches for mitigating ...
Comments
Information & Contributors
Information
Published In
October 2024
1438 pages
ISBN:9798400705052
DOI:10.1145/3640457
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives International 4.0 License.
Sponsors
- SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web
- SIGAI: ACM Special Interest Group on Artificial Intelligence
- SIGKDD: ACM Special Interest Group on Knowledge Discovery in Data
- SIGIR: ACM Special Interest Group on Information Retrieval
- SIGCHI: ACM Special Interest Group on Computer-Human Interaction
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 08 October 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
Acceptance Rates
Overall Acceptance Rate 254 of 1,295 submissions, 20%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 584Total Downloads
- Downloads (Last 12 months)584
- Downloads (Last 6 weeks)155
Reflects downloads up to 12 Jan 2025
Other Metrics
Citations
Cited By
View all- Dinnissen K(2024)Fairness and Transparency in Music Recommender Systems: Improvements for ArtistsProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688024(1368-1375)Online publication date: 8-Oct-2024
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in