research-article

Should I Follow the Crowd?: A Probabilistic Analysis of the Effectiveness of Popularity in Recommender Systems

Authors:

Rocío Cañamares,

Pablo CastellsAuthors Info & Claims

SIGIR '18: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval

Pages 415 - 424

https://doi.org/10.1145/3209978.3210014

Published: 27 June 2018 Publication History

Abstract

The use of IR methodology in the evaluation of recommender systems has become common practice in recent years. IR metrics have been found however to be strongly biased towards rewarding algorithms that recommend popular items "the same bias that state of the art recommendation algorithms display. Recent research has confirmed and measured such biases, and proposed methods to avoid them. The fundamental question remains open though whether popularity is really a bias we should avoid or not; whether it could be a useful and reliable signal in recommendation, or it may be unfairly rewarded by the experimental biases. We address this question at a formal level by identifying and modeling the conditions that can determine the answer, in terms of dependencies between key random variables, involving item rating, discovery and relevance. We find conditions that guarantee popularity to be effective or quite the opposite, and for the measured metric values to reflect a true effectiveness, or qualitatively deviate from it. We exemplify and confirm the theoretical findings with empirical results. We build a crowdsourced dataset devoid of the usual biases displayed by common publicly available data, in which we illustrate contradictions between the accuracy that would be measured in a common biased offline experimental setting, and the actual accuracy that can be measured with unbiased observations.

References

[1]

P. Adamopoulos and A. Tuzhilin. 2014. On unexpectedness in recommender systems: or how to better expect the unexpected. ACM Transactions on Intelligent Systems and Technology 5, 4, (Jan. 2014). ACM, New York, NY, 1--32.

Digital Library

[2]

G. Adomavicius and A. Tuzhilin. 2005. Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE TKDE 17, 6 (June 2005). IEEE, Piscataway, NJ, USA, 734--749.

Digital Library

[3]

A. Bandura. 1971. Social Learning Theory. General Learning Press, New York.

[4]

A. Bellogín, P. Castells, and I. Cantador. 2017. Statistical Biases in Information Retrieval Metrics for Recommender Systems. Information Retrieval 20, 6 (Jul. 2017). Springer, Dordrecht, Netherlands, 606--634.

Digital Library

[5]

S. Bikhchandani, D. Hirshleifer, and I. Welch. 1992. A Theory of Fads, Custom, and Cultural Change as Informational Cascades. The Journal of Political Economy 100, 5 (Oct. 1992). University of Chicago Press, Chicago, IL, USA, 992--1026.

[6]

R. Bredereck and E. Elkind. 2017. Manipulating Opinion Diffusion in Social Networks. In Proc. of the 26th International Joint Conference on Artificial Intelligence (IJCAI 2017). Morgan Kaufmann Publishers, San Francisco, CA, USA 894--900.

Digital Library

[7]

J. Bryant and M. B. Oliver (Eds.). 2008. Media Effects: Advances in Theory and Research, 3rd edition. Routledge, Abingdon, UK.

[8]

R. Cañamares and P. Castells. 2014. Exploring social network effects on popularity biases in recommender systems. In 6th ACM RecSys Workshop on Recommender Systems and the Social Web (RSWeb 2014). Foster City, CA, Oct. 2014.

[9]

R. Cañamares and P. Castells. 2017. A Probabilistic Reformulation of Memory-Based Collaborative Filtering -- Implications on Popularity Biases. In Proc. of the 40th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2017). ACM, New York, USA, 215--224.

Digital Library

[10]

R. Cañamares and P. Castells. 2017. On the Optimal Non-Personalized Recommendation: From the PRP to the Discovery False Negative Principle. ACM SIGIR Workshop on Axiomatic Thinking for Information Retrieval and Related Tasks (ATIR 2017). Tokyo, Japan, Aug. 2017.

[11]

P. Castells, N. J. Hurley, S. Vargas. 2015. Novelty and Diversity in Recommender Systems. In: Recommender Systems Handbook, 2nd edition, F. Ricci, L. Rokach, and B. Shapira (Eds.). Springer, New York, NY, USA, 881--918.

[12]

O. Celma and P. Herrera. 2008. A new approach to evaluating novel recommendations. In Proc. of the 2nd ACM Conference on Recommender Systems (RecSys 2008). ACM, New York, NY, USA, 179--186.

Digital Library

[13]

R. B. Cialdini and N. J. Goldstein. 2004. Social Influence: Compliance and Conformity. Annual Review of Psychology 55 (Feb. 2004). Palo Alto, CA, USA, 591--621.

[14]

P. Cremonesi, Y. Koren, and R. Turrin. 2010. Performance of recommender algorithms on top-n recommendation tasks. In Proc. of the 4th ACM Conference on Recommender Systems (RecSys 2010). ACM, New York, NY, USA, 39--46.

Digital Library

[15]

D. Fleder and K. Hosanagar. 2009. Blockbuster culture's next rise or fall: The impact of recommender systems on sales diversity. Management Science 55, 5 (May 2009). Informs, Catonsville, MD, USA, 697--712.

Digital Library

[16]

S. Goel, A. Broder, E. Gabrilovich, and B. Pang. 2010. Anatomy of the long tail: ordinary people with extraordinary tastes. In Proc. of the 3rd ACM Int. Conf. on Web Search and Data Mining (WSDM 2010). ACM, New York, NY, USA, 201--210.

Digital Library

[17]

F. M. Harper and J. A. Konstan. 2016. The MovieLens Datasets: History and Context. ACM TOIS 5, 4 (Jan. 2016). ACM, New York, NY, USA.

Digital Library

[18]

F. M. Harper, X. Li, Y. Chen and J. A. Konstan. 2005. An Economic Model of User Rating in an Online Recommender System. In Proc. of the 10th International Conference on User Modeling (UM 2005). Springer, Berlin, Germany, 307--316.

Digital Library

[19]

H. He and E. A. Garcia. 2009. Learning from Imbalanced Data. IEEE TKDE 21, 9 (Sept.2009). IEEE, Piscataway, NJ, USA, 1263--1284.

Digital Library

[20]

Y. Hu, Y. Koren, and C. Volinsky. 2008. Collaborative Filtering for Implicit Feedback Datasets. In Proc. of the 8th IEEE International Conference on Data Mining (ICDM 2008). IEEE Computer Society, Washington, DC, USA, 15--19.

Digital Library

[21]

D. Jannach, L. Lerche, I. Kamehkhosh, and M. Jugovac. 2015. What recommenders recommend: an analysis of recommendation biases and possible countermeasures. User Modeling and User-Adapted Interaction 25, 5 (Dec. 2015). Kluwer Academic Publishers, Hingham, MA, USA, 427--491.

Digital Library

[22]

G. Linden, B. Smith, and J. York. 2003. Amazon.com Recommendations: Item-to-Item Collaborative Filtering. IEEE Internet Computing 7, 1 (Jan. 2003). IEEE, Piscataway, NJ, USA, 76--80.

Digital Library

[23]

R. J. A. Little and D. B. Rubin. 1987. Statistical analysis with missing data. John Wiley&Sons, Hoboken, NJ, USA.

Digital Library

[24]

B. M. Marlin, R. S. Zemel, S. T. Roweis, and M. Slaney. 2007. Collaborative Filtering and the Missing at Random Assumption. In Proc. of the 23rd Conf. on Uncertainty in Artificial Intelligence (UAI 2007). AUAI Press, Arlington, VA, 267--275.

Digital Library

[25]

B. Marlin and R. Zemel. 2009. Collaborative prediction and ranking with non-random missing data. In Proc. of the 3rd ACM Conference on Recommender Systems (RecSys 2009). ACM, New York, NY, USA, 5--12.

Digital Library

[26]

M. Moussaïd, J. E. Kämmer, P. P. Analytis, and H. Neth. 2013. Social Influence and the Collective Dynamics of Opinion Formation. PLoS One 8, 11 (Nov. 2013). Public Library of Science, San Francisco, CA, USA.

[27]

S. A. Myers, C. Zhu, and J. Leskovec. 2012. Information diffusion and external influence in networks. In Proc of the 18th ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining (KDD 2012). ACM, New York, NY, USA, 33--41.

Digital Library

[28]

E. Pariser. 2012. The Filter Bubble: How the New Personalized Web Is Changing What We Read and How We Think. Penguin Books, London, UK.

Digital Library

[29]

J. Ratkiewicz, S. Fortunato, A. Flammini, F. Menczer and A. Vespignani. 2010. Characterizing and Modeling the Dynamics of Online Popularity. Physical Review Letters 105, 15 (Oct. 2010). APS, Ridge, NY, USA.

[30]

S. E. Robertson. 1977. The Probability Ranking in IR. Journal of Documentation 33, 4 (Jan. 1977), 294--304.

[31]

M. J. Salganik, P. S. Dodds and D. J. Watts. 2006. Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market. Science 311, 5762 (Feb. 2006). AAAS, Washington, D.C., USA, 854--856.

[32]

G. Shani and A. Gunawardana, 2015. Evaluating Recommendation Systems. In: Recommender Systems Handbook, 2nd edition, F. Ricci, L. Rokach, and B. Shapira (Eds.). Springer, New York, NY, USA, 265--308.

[33]

A. Sinha, D. F. Gleich, and K. Ramani. 2016. Deconvolving Feedback Loops in Recommender Systems. In Proc. of the 30th Annual Conference on Neural Information Processing Systems (NIPS 2016). Barcelona, Spain, Dec. 2016, 3243--3251.

Digital Library

[34]

H. Steck. 2010. Training and testing of recommender systems on data missing not at random. In Proc. of the 16th ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining (KDD 2010). ACM, New York, NY, USA, 713--722.

Digital Library

[35]

H. Steck. 2011. Item popularity and recommendation accuracy. In Proc. of the 5th ACM Conference on Recommender Systems (RecSys 2011). ACM, New York, NY, USA, 125--132.

Digital Library

[36]

C. Zhai and, J. D. Lafferty. 2004. A study of smoothing methods for language models applied to information retrieval. ACM Transactions on Information Systems 22, 2 (April 2004). ACM, New York, NY, USA, 179--14.

Digital Library

Cited By

Li SHu XGuo JLiu BQi MJia Y(2024)Popularity-Debiased Graph Self-Supervised for RecommendationElectronics10.3390/electronics1304067713:4(677)Online publication date: 6-Feb-2024
https://doi.org/10.3390/electronics13040677
Bompotas ATriantafyllopoulos PRaptis GKatsini CMakris CBoratto LGena CMarras MGermanakos PPopescus E(2024)Towards Exploring Personalized Hyperlink Recommendations Through Machine LearningAdjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3631700.3664913(528-533)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3631700.3664913
Bauer CZangerle ESaid A(2024)Exploring the Landscape of Recommender Systems Evaluation: Practices and PerspectivesACM Transactions on Recommender Systems10.1145/36291702:1(1-31)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1145/3629170
Show More Cited By

Index Terms

Should I Follow the Crowd?: A Probabilistic Analysis of the Effectiveness of Popularity in Recommender Systems
1. Information systems
  1. Information retrieval
  2. Information systems applications
    1. Data mining
      1. Collaborative filtering

Recommendations

Analyzing weighting schemes in collaborative filtering: cold start, post cold start and power users
SAC '12: Proceedings of the 27th Annual ACM Symposium on Applied Computing

Collaborative filtering recommender systems provide their users with relevant items based on information from other similar users. Popular collaborative filtering approaches such as Pearson correlation coefficient and cosine similarity, compute the ...
A Novel Framework for Improving Recommender Diversity
International Workshop on Behavior and Social Informatics on Behavior and Social Computing - Volume 8178

Recommender systems are being used to assist users in finding relevant items from a large set of alternatives in many online applications. However, while most research up to this point has focused on improving the accuracy of recommender systems, other ...
A new approach to evaluating novel recommendations
RecSys '08: Proceedings of the 2008 ACM conference on Recommender systems

This paper presents two methods, named Item- and User-centric, to evaluate the quality of novel recommendations. The former method focuses on analyzing the item-based recommendation network. The aim is to detect whether the network topology has any ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '18: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval

June 2018

1509 pages

ISBN:9781450356572

DOI:10.1145/3209978

General Chairs:
Kevyn Collins-Thompson
University of Michigan, United States
,
Qiaozhu Mei
University of Michigan, United States
,
Program Chairs:
Brian Davison
Lehigh University, United States
,
Yiqun Liu
Tsinghua University, China
,
Emine Yilmaz
University College London, United Kingdom

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 June 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Author Tags

Qualifiers

Research-article

Funding Sources

Ministerio de Economia y Competitividad Spain

Conference

SIGIR '18

Sponsor:

SIGIR

SIGIR '18: The 41st International ACM SIGIR conference on research and development in Information Retrieval

July 8 - 12, 2018

MI, Ann Arbor, USA

Acceptance Rates

SIGIR '18 Paper Acceptance Rate 86 of 409 submissions, 21%;

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

58
Total Citations
View Citations
1,800
Total Downloads

Downloads (Last 12 months)89
Downloads (Last 6 weeks)1

Reflects downloads up to 06 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li SHu XGuo JLiu BQi MJia Y(2024)Popularity-Debiased Graph Self-Supervised for RecommendationElectronics10.3390/electronics1304067713:4(677)Online publication date: 6-Feb-2024
https://doi.org/10.3390/electronics13040677
Bompotas ATriantafyllopoulos PRaptis GKatsini CMakris CBoratto LGena CMarras MGermanakos PPopescus E(2024)Towards Exploring Personalized Hyperlink Recommendations Through Machine LearningAdjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3631700.3664913(528-533)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3631700.3664913
Bauer CZangerle ESaid A(2024)Exploring the Landscape of Recommender Systems Evaluation: Practices and PerspectivesACM Transactions on Recommender Systems10.1145/36291702:1(1-31)Online publication date: 7-Mar-2024
https://dl.acm.org/doi/10.1145/3629170
Fernandez MBellogín ACantador I(2024)Analysing the Effect of Recommendation Algorithms on the Spread of MisinformationProceedings of the 16th ACM Web Science Conference10.1145/3614419.3644003(159-169)Online publication date: 21-May-2024
https://dl.acm.org/doi/10.1145/3614419.3644003
Liao JZhou WLuo FWen J(2024)Enhancing Disentanglement of Popularity Bias for Recommendation With Triplet Contrastive LearningIEEE Transactions on Services Computing10.1109/TSC.2024.337892517:3(921-933)Online publication date: May-2024
https://doi.org/10.1109/TSC.2024.3378925
Sonoda AToriumi FNakajima H(2024)User Experiments on the Effect of the Diversity of Consumption on News ServicesIEEE Access10.1109/ACCESS.2024.336777012(31841-31852)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3367770
Salganik RDiaz FFarnadi G(2024)Fairness Through Domain Awareness: Mitigating Popularity Bias for Music DiscoveryAdvances in Information Retrieval10.1007/978-3-031-56066-8_27(351-368)Online publication date: 24-Mar-2024
https://dl.acm.org/doi/10.1007/978-3-031-56066-8_27
Cakmak MOkeke OOnyepunuka USpann BAgarwal N(2024)Investigating Bias in YouTube Recommendations: Emotion, Morality, and Network Dynamics in China-Uyghur ContentComplex Networks & Their Applications XII10.1007/978-3-031-53468-3_30(351-362)Online publication date: 20-Feb-2024
https://doi.org/10.1007/978-3-031-53468-3_30
Zhang SZhang NWang WLiu QLi J(2023)A Social Recommendation Model Based on Basic Spatial Mapping and Bilateral Generative Adversarial NetworksEntropy10.3390/e2510138825:10(1388)Online publication date: 28-Sep-2023
https://doi.org/10.3390/e25101388
Zhang AMa WZheng JWang XChua T(2023)Robust Collaborative Filtering to Popularity Distribution ShiftACM Transactions on Information Systems10.1145/3627159Online publication date: 12-Oct-2023
https://dl.acm.org/doi/10.1145/3627159
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents