research-article

Open access

Calibrated recommendations

Author:

Harald SteckAuthors Info & Claims

RecSys '18: Proceedings of the 12th ACM Conference on Recommender Systems

Pages 154 - 162

https://doi.org/10.1145/3240323.3240372

Published: 27 September 2018 Publication History

Abstract

When a user has watched, say, 70 romance movies and 30 action movies, then it is reasonable to expect the personalized list of recommended movies to be comprised of about 70% romance and 30% action movies as well. This important property is known as calibration, and recently received renewed attention in the context of fairness in machine learning. In the recommended list of items, calibration ensures that the various (past) areas of interest of a user are reflected with their corresponding proportions. Calibration is especially important in light of the fact that recommender systems optimized toward accuracy (e.g., ranking metrics) in the usual offline-setting can easily lead to recommendations where the lesser interests of a user get crowded out by the user's main interests-which we show empirically as well as in thought-experiments. This can be prevented by calibrated recommendations. To this end, we outline metrics for quantifying the degree of calibration, as well as a simple yet effective re-ranking algorithm for post-processing the output of recommender systems.

Supplementary Material

MP4 File (p154-steck.mp4)

Download
2701.47 MB

References

[1]

A. Ashkan, B. Kveton, S. Berkovsky, and Z. Wen. 2014. Diversified Utility Maximization for Recommendations. In ACM Conference on Recommender Systems (RecSys).

[2]

A. Ashkan, B. Kveton, S. Berkovsky, and Z. Wen. 2015. Optimal Greedy Diversity for Recommendation. In Int. Joint Conf. on Artificial Intelligence (IJCAI).

Digital Library

[3]

J. Bennet and S. Lanning. 2007. The Netflix Prize. In Workshop at SIGKDD-07, ACM Conference on Knowledge Discovery and Data Mining.

[4]

A. Bhaskara, M. Ghadiri, and V. Mirrokni. 2016. Linear Relaxations for Finding Diverse Elements in Metric Spaces. In Advances in Neural Information Processing Systems (NIPS).

Digital Library

[5]

D.M. Blei, A.Y. Ng, and M.Jordan. 2001. Latent Dirichlet Allocation. In Advances in Neural Information Processing Systems (NIPS).

Digital Library

[6]

J. Carbonell and J. Goldstein. 1998. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In ACM Conference on Research and development in information retrieval (SIGIR).

Digital Library

[7]

O. Chapelle, D. Metzler, Y. Zhang, and P. Grinspan. 2009. Expected Reciprocal Rank for Graded Relevance. In CIKM.

Digital Library

[8]

L. Chen, G. Zhang, and H. Zhou. 2017. Improving the Diversity ofTop-N Recommendation via Determinantal Point Process. arXiv:1709.05135.

[9]

V. Dang and W. B. Croft. 2012. Diversity by Proportionality: An Election-based Approach to Search Result Diversification. In ACM Conference on Research and development in information retrieval (SIGIR).

Digital Library

[10]

D.P. Foster and R.V. Vohra. 1998. Asymptotic calibration. Biometrika 85 (1998), 379--90. Issue 2.

[11]

M. Gartrell, U. Paquet, and N. Koenigstein. 2016. Bayesian low-rank determinantal point processes. In ACM Conference on Recommender Systems (RecSys). 349--56.

Digital Library

[12]

M. Hardt, E. Price, and N. Srebro. 2016. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems (NIPS).

Digital Library

[13]

F. M. Harper and J. A. Konstan. 2015. The MovieLens Datasets: History and Context. ACM Transactions on Interactive Intelligent Systems (TiiS) 5 (2015). Issue 4.

Digital Library

[14]

Y. Hu, Y. Koren, and C. Volinsky. 2008. Collaborative Filtering for Implicit Feedback Datasets. In IEEE International Conference on Data Mining (ICDM).

Digital Library

[15]

Neil Hurley and Mi Zhang. 2011. Novelty and diversity in top-N recommendation-analysis and evaluation. ACM Transactions on Internet Technology (TOIT) 10 (2011). Issue 4.

Digital Library

[16]

J. Kleinberg, S. Mullainathan, and M. Raghavan. 2016. Inherent Trade-Offs in the Fair Determination of Risk Scores. In Advances in Neural Information Processing Systems (NIPS).

[17]

G. L. Nemhauser and L.A. Wolsey. 1978. An Analysis of Approximations for Maximizing Submodular Set Functions. Mathematical Programming 14 (1978).

Digital Library

[18]

R. Pan, Y. Zhou, B. Cao, N. Liu, R. Lukose, M. Scholz, and Q. Yang. 2008. One-Class Collaborative Filtering. In IEEE International Conference on Data Mining (ICDM).

[19]

L. Qin and X. Zhu. 2013. Promoting Diversity in Recommendation by Entropy Regularizer. In Int. Joint Conf. on Artificial Intelligence (IJCAI).

Digital Library

[20]

R. L. T. Santos, C. Macdonald, and I. Ounis. 2010. Exploiting Query Reformulation for Web Search Result Diversification. In International World Wide Web Conference (WWW).

Digital Library

[21]

G. Shani and A. Gunawardana. 2011. Evaluating Recommendation systems. In Recommender Systems Handbook. Springer, 257--97.

[22]

Y. Shinohara. 2014. A submodular optimization approach to sentence set selection. In IEEE International Conference on Acoustic, Speech and Signal processing (ICASSP).

[23]

H. Steck. 2010. Training and Testing of Recommender Systems on Data Missing Not at Random. In ACM Conference on Knowledge Discovery and Data Mining (KDD). 713--22.

Digital Library

[24]

A. Swaminathan, A. Krishnamurthy, A. Agarwal, M. Dudik, J. Langford, D. Jose, and I. Zitouni. 2017. Off-policy evaluation for slate recommendation. In Advances in Neural Information Processing Systems (NIPS). 3635--45.

[25]

C. H. Teo, H. Nassif, D. Hill, S. Srinivasan, M. Goodman, V. Mohan, and S. V. N. Vishwanathan. 2016. Adaptive, Personalized Diversity for Visual Discovery. In ACM Conference on Recommender Systems (RecSys). 35--8.

Digital Library

[26]

S. Vargas, L. Baltrunas, A. Karatzoglou, and P. Castells. 2014. Coverage, Redundancy and Size-Awareness in Genre Diversity for Recommender Systems. In ACM Conference on Recommender Systems (RecSys).

Digital Library

[27]

S. Vargas, P. Castells, and D. Vallet. 2012. Explicit Relevance Models in Intent-Oriented Information Retrieval Diversification. In ACM Conference on Research and Development in Information Retrieval (SIGIR).

Digital Library

[28]

B. Woodworth, S. Gunasekar, M.I. Ohannessian, and N. Srebro. 2017. Learning Non-Discriminatory Predictors. arXiv:1702.06081.

[29]

S. Yao and B. Huang. 2017. Beyond Parity: Fairness Objectives for Collaborative Filtering. In Advances in Neural Information Processing Systems (NIPS).

[30]

B. Zadrozny and C. Elkan. 2001. Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. In International Conference on Machine Learning (ICML). 609--16.

Digital Library

[31]

M. Zhang and N. Hurley. 2008. Avoiding monotony: improving the diversity of recommendation lists. In ACM Conference on Recommender Systems (RecSys). 123--30.

Digital Library

[32]

C.-N. Ziegler, S. M. McNee, J. A. Konstan, and G. Lausen. 2005. Improving recommendation lists through topic diversification. In International World Wide Web Conference (WWW). 22--32.

Digital Library

[33]

I. Zliobaite. 2015. A survey on measuring indirect discrimination in machine learning. arXiv:1511.00148.

Cited By

Guo WCui YZheng K(2025)IntFair:Graph Neural Networks for Fair Recommendations with Interest AwarenessDatabase Systems for Advanced Applications10.1007/978-981-97-5555-4_1(3-18)Online publication date: 12-Jan-2025
https://doi.org/10.1007/978-981-97-5555-4_1
Li SHu XGuo JLiu BQi MJia Y(2024)Popularity-Debiased Graph Self-Supervised for RecommendationElectronics10.3390/electronics1304067713:4(677)Online publication date: 6-Feb-2024
https://doi.org/10.3390/electronics13040677
Zheng YChen ZQin JLin LWooldridge MDy JNatarajan S(2024)FacetCRSProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i8.28794(9405-9413)Online publication date: 20-Feb-2024
https://dl.acm.org/doi/10.1609/aaai.v38i8.28794
Show More Cited By

Index Terms

Calibrated recommendations
1. Information systems
  1. Information systems applications
    1. Data mining
      1. Collaborative filtering

Recommendations

A comparison of calibrated and intent-aware recommendations
RecSys '19: Proceedings of the 13th ACM Conference on Recommender Systems

Calibrated and intent-aware recommendation are recent approaches to recommendation that have apparent similarities. Both try, to a certain extent, to cover the user's interests, as revealed by her user profile. In this paper, we compare them in detail. ...
Calibrated Recommendations as a Minimum-Cost Flow Problem
WSDM '23: Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining

Calibration in recommender systems has recently gained significant attention. In the recommended list of items, calibration ensures that the various (past) areas of interest of a user are reflected with their corresponding proportions. For instance, if a ...
A framework for diversifying recommendation lists by user interest expansion

Recommender systems have been widely used to discover users' preferences and recommend interesting items to users during this age of information overload. Researchers in the field of recommender systems have realized that the quality of a top-N ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

RecSys '18: Proceedings of the 12th ACM Conference on Recommender Systems

September 2018

600 pages

ISBN:9781450359016

DOI:10.1145/3240323

General Chairs:
Sole Pera
Boise State University
,
Michael Ekstrand
Boise State University
,
Program Chairs:
Xavier Amatriain
Curai, USA
,
John O'Donovan
University of California, Santa Barbara

Copyright © 2018 Owner/Author.

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike International 4.0 License.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 September 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

RecSys '18

Sponsor:

SIGCHI

RecSys '18: Twelfth ACM Conference on Recommender Systems

October 2, 2018

British Columbia, Vancouver, Canada

Acceptance Rates

RecSys '18 Paper Acceptance Rate 32 of 181 submissions, 18%;

Overall Acceptance Rate 254 of 1,295 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

153
Total Citations
View Citations
16,876
Total Downloads

Downloads (Last 12 months)3,823
Downloads (Last 6 weeks)351

Reflects downloads up to 12 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Guo WCui YZheng K(2025)IntFair:Graph Neural Networks for Fair Recommendations with Interest AwarenessDatabase Systems for Advanced Applications10.1007/978-981-97-5555-4_1(3-18)Online publication date: 12-Jan-2025
https://doi.org/10.1007/978-981-97-5555-4_1
Li SHu XGuo JLiu BQi MJia Y(2024)Popularity-Debiased Graph Self-Supervised for RecommendationElectronics10.3390/electronics1304067713:4(677)Online publication date: 6-Feb-2024
https://doi.org/10.3390/electronics13040677
Zheng YChen ZQin JLin LWooldridge MDy JNatarajan S(2024)FacetCRSProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i8.28794(9405-9413)Online publication date: 20-Feb-2024
https://dl.acm.org/doi/10.1609/aaai.v38i8.28794
Zhao YWang YLiu YCheng XAggarwal CDerr T(2024)Fairness and Diversity in Recommender Systems: A SurveyACM Transactions on Intelligent Systems and Technology10.1145/366492816:1(1-28)Online publication date: 21-May-2024
https://dl.acm.org/doi/10.1145/3664928
Zheng YWang GLiu YLin LCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Diversity Matters: User-Centric Multi-Interest Learning for Conversational Movie RecommendationProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680909(9515-9524)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680909
Naghiaei MDehghan MRahmani HAzizi JAliannejadi MOosterhuis HBast HXiong C(2024)Personalized Beyond-accuracy Calibration in RecommendationProceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3664190.3672507(107-116)Online publication date: 2-Aug-2024
https://dl.acm.org/doi/10.1145/3664190.3672507
Ovaisi ZSaadatpanah PSefati SOhannessian MZheleva E(2024)Fairness of Interaction in Ranking under Position, Selection, and Trust BiasACM Transactions on Recommender Systems10.1145/36528643:2(1-28)Online publication date: 6-Apr-2024
https://dl.acm.org/doi/10.1145/3652864
Yang HWu XQiu ZZheng YChen X(2024)Distributional Fairness-aware RecommendationACM Transactions on Information Systems10.1145/365285442:5(1-28)Online publication date: 29-Apr-2024
https://dl.acm.org/doi/10.1145/3652854
Ahn YLin Y(2024)Break Out of a Pigeonhole: A Unified Framework for Examining Miscalibration, Bias, and Stereotype in Recommender SystemsACM Transactions on Intelligent Systems and Technology10.1145/365004415:4(1-20)Online publication date: 29-Feb-2024
https://dl.acm.org/doi/10.1145/3650044
Li NKang BLijffijt JDe Bie T(2024)FEIR: Quantifying and Reducing Envy and Inferiority for Fair Recommendation of Limited ResourcesACM Transactions on Intelligent Systems and Technology10.1145/364389115:4(1-24)Online publication date: 3-Feb-2024
https://dl.acm.org/doi/10.1145/3643891
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents