research-article

Offline and online evaluation of news recommender systems at swissinfo.ch

Authors:

Florent Garcin,

Olivier Donatsch,

Christophe Bruttin,

Amr HuberAuthors Info & Claims

RecSys '14: Proceedings of the 8th ACM Conference on Recommender systems

Pages 169 - 176

https://doi.org/10.1145/2645710.2645745

Published: 06 October 2014 Publication History

Abstract

We report on the live evaluation of various news recommender systems conducted on the website swissinfo.ch. We demonstrate that there is a major difference between offline and online accuracy evaluations. In an offline setting, recommending most popular stories is the best strategy, while in a live environment this strategy is the poorest. For online setting, context-tree recommender systems which profile the users in real-time improve the click-through rate by up to 35%. The visit length also increases by a factor of 2.5. Our experience holds important lessons for the evaluation of recommender systems with offline data as well as for the use of the click-through rate as a performance indicator.

Supplementary Material

MP4 File (p169-sidebyside.mp4)

Download
47.45 MB

References

[1]

J. Ahn, P. Brusilovsky, J. Grady, and D. He. Open user profiles for adaptive news systems: help or harm? In Conf. on WWW, pages 11--20, 2007.

Digital Library

[2]

D. Billsus and M. Pazzani. A hybrid user model for news story classification. In Conf. on User Modeling, pages 99--108, 1999.

Digital Library

[3]

R. Burke. Hybrid recommender systems: Survey and experiments. User Modeling and User-Adapted Interaction, 12:331--370, November 2002.

Digital Library

[4]

L. Chen and P. Pu. Eye-tracking study of user behavior in recommender interfaces. In User Modeling, Adaptation, and Personalization, pages 375--380, 2010.

Digital Library

[5]

P. Cremonesi, Y. Koren, and R. Turrin. Performance of recommender algorithms on top-n recommendation tasks. In Conf. on Rec. Systems, pages 39--46, 2010.

Digital Library

[6]

A. Das, M. Datar, A. Garg, and S. Rajaram. Google news personalization: scalable online collaborative filtering. In Conf. on WWW, pages 271--280, 2007.

Digital Library

[7]

F. Garcin, C. Dimitrakakis, and B. Faltings. Personalized news recommendation with context trees. In Conf. on Rec. Systems, pages 105--112, 2013.

Digital Library

[8]

F. Garcin and B. Faltings. Pen recsys: A personalized news recommender systems framework. In Conf. on Rec. Systems, pages 469--470, 2013.

Digital Library

[9]

J. Herlocker, J. Konstan, L. Terveen, and J. Riedl. Evaluating collaborative filtering recommender systems. Trans. on Info. Systems, 22:5--53, 2004.

Digital Library

[10]

W. IJntema, F. Goossen, F. Frasincar, and F. Hogenboom. Ontology-based news recommendation. In Workshop on Data Semantics, page 16, 2010.

Digital Library

[11]

B. Kille, F. Hopfgartner, T. Brodt, and T. Heintz. The plista dataset. In News Recommender Systems Workshop, pages 16--23, 2013.

Digital Library

[12]

E. Kirshenbaum, G. Forman, and M. Dugan. A live comparison of methods for personalized article recommendation at forbes.com. In ECML - PKDD, pages 51--66, 2012.

Digital Library

[13]

K. Lerman and T. Hogg. Using a model of social dynamics to predict popularity of news. In Conf. on WWW, pages 621--630, 2010.

Digital Library

[14]

L. Li, W. Chu, J. Langford, and R. Schapire. A contextual-bandit approach to personalized news article recommendation. In Conf. on WWW, pages 661--670, 2010.

Digital Library

[15]

J. Liu, P. Dolan, and E. Pedersen. Personalized news recommendation based on click behavior. In Proc. of IUI, pages 31--40, 2010.

Digital Library

[16]

P. Pu, L. Chen, and R. Hu. A user-centric evaluation framework for recommender systems. In Conf. on Rec. Systems, pages 157--164, 2011.

Digital Library

[17]

P. Resnick, N. Iacovou, M. Suchak, P. Bergstrom, and J. Riedl. Grouplens: an open architecture for collaborative filtering of netnews. In CSCW, pages 175--186, 1994.

Digital Library

[18]

A. Said, A. Bellogin, J. Lin, and A. de Vries. Do recommendations matter?: News recommendation in real life. In CSCW, pages 237--240, 2014.

Digital Library

[19]

A. Said, B. Fields, B. Jain, and S. Albayrak. User-centric evaluation of a k-furthest neighbor collaborative filtering recommender algorithm. In CSCW, pages 1399--1408, 2013.

Digital Library

[20]

A. Said, J. Lin, A. Bellogin, and A. de Vries. A month in the life of a production news recommender system. In Workshop on Living Labs for Information Retrieval Evaluation, pages 7--10, 2013.

Digital Library

[21]

G. Shani and A. Gunawardana. Evaluating recommendation systems. In Recommender Systems Handbook, pages 257--297. Springer, 2011.

[22]

M. Tavakolifard, J. Gulla, K. Almeroth, F. Hopfgartner, B. Kille, T. Plumbaum, A. Lommatzsch, T. Brodt, A. Bucko, and T. Heintz. Workshop and challenge on news recommender systems. In Conf. on Rec. Systems, pages 481--482, 2013.

Digital Library

[23]

H. Zheng, D. Wang, Q. Zhang, H. Li, and T. Yang. Do clicks measure recommendation relevancy?: an empirical user study. In Conf. on Rec. Systems, pages 249--252, 2010.

Digital Library

Cited By

Tavakoli LTrippas JZamani HScholer FSanderson M(2024)Online and Offline Evaluation in Search ClarificationACM Transactions on Information Systems10.1145/368178643:1(1-30)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3681786
Bauer CBagchi CHundogan Ovan Es K(2024)Where Are the Values? A Systematic Literature Review on News Recommender SystemsACM Transactions on Recommender Systems10.1145/36548052:3(1-40)Online publication date: 28-Mar-2024
https://dl.acm.org/doi/10.1145/3654805
Moreschini SLenarduzzi VCoba LCodabux ZGalster MSpinola R(2024)Towards a Technical Debt for AI-based Recommender SystemProceedings of the 7th ACM/IEEE International Conference on Technical Debt10.1145/3644384.3648574(36-39)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3644384.3648574
Show More Cited By

Index Terms

Offline and online evaluation of news recommender systems at swissinfo.ch
1. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Document filtering
      2. Information extraction

Recommendations

A comparative analysis of offline and online evaluations and discussion of research paper recommender system evaluation
RepSys '13: Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation

Offline evaluations are the most common evaluation method for research paper recommender systems. However, no thorough discussion on the appropriateness of offline evaluations has taken place, despite some voiced criticism. We conducted a study in which ...
Real-time news recommender system
ECMLPKDD'10: Proceedings of the 2010th European Conference on Machine Learning and Knowledge Discovery in Databases - Volume Part III

In this demo we present a robust system for delivering real-time news recommendation to the user based on the user's history of the past visits to the site, current user's context and popularity of stories. Our system is running live providing real-time ...
Real-time news recommender system
ECML PKDD'10: Proceedings of the 2010 European conference on Machine learning and knowledge discovery in databases: Part III

In this demo we present a robust system for delivering real-time news recommendation to the user based on the user's history of the past visits to the site, current user's context and popularity of stories. Our system is running live providing real-time ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

RecSys '14: Proceedings of the 8th ACM Conference on Recommender systems

October 2014

458 pages

ISBN:9781450326681

DOI:10.1145/2645710

General Chairs:
Alfred Kobsa
University of California, Irvine
,
Michelle Zhou
IBM
,
Program Chairs:
Martin Ester
Simon Fraser University
,
Yehuda Koren
Google

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 October 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

RecSys'14

Sponsor:

SIGCHI

RecSys'14: Eighth ACM Conference on Recommender Systems

October 6 - 10, 2014

California, Foster City, Silicon Valley, USA

Acceptance Rates

RecSys '14 Paper Acceptance Rate 35 of 234 submissions, 15%;

Overall Acceptance Rate 254 of 1,295 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

118
Total Citations
View Citations
1,527
Total Downloads

Downloads (Last 12 months)52
Downloads (Last 6 weeks)12

Reflects downloads up to 16 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Tavakoli LTrippas JZamani HScholer FSanderson M(2024)Online and Offline Evaluation in Search ClarificationACM Transactions on Information Systems10.1145/368178643:1(1-30)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3681786
Bauer CBagchi CHundogan Ovan Es K(2024)Where Are the Values? A Systematic Literature Review on News Recommender SystemsACM Transactions on Recommender Systems10.1145/36548052:3(1-40)Online publication date: 28-Mar-2024
https://dl.acm.org/doi/10.1145/3654805
Moreschini SLenarduzzi VCoba LCodabux ZGalster MSpinola R(2024)Towards a Technical Debt for AI-based Recommender SystemProceedings of the 7th ACM/IEEE International Conference on Technical Debt10.1145/3644384.3648574(36-39)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3644384.3648574
Vandenbroucke HSmets A(2024)It's (not) all about that CTR: A Multi-Stakeholder Perspective on News Recommender MetricsProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688183(999-1003)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3640457.3688183
Jeng J(2024)Bridging Viewpoints in News with Recommender SystemsProceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3688008(1283-1289)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3640457.3688008
Kille BLommatzsch ATreuillier CYadav VÖzgöbek Ö(2024)12th International Workshop on News Recommendation and Analytics (INRA'24)Proceedings of the 18th ACM Conference on Recommender Systems10.1145/3640457.3687100(1258-1261)Online publication date: 8-Oct-2024
https://dl.acm.org/doi/10.1145/3640457.3687100
Jeunen OPotapov IUstimenko ABaeza-Yates RBonchi F(2024)On (Normalised) Discounted Cumulative Gain as an Off-Policy Evaluation Metric for Top-n RecommendationProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671687(1222-1233)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671687
Rosnes DStarke ATrattner C(2024)Shaping the Future of Content-based News Recommenders: Insights from Evaluating Feature-Specific Similarity MetricsProceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3627043.3659560(201-211)Online publication date: 22-Jun-2024
https://dl.acm.org/doi/10.1145/3627043.3659560
Starke ASolberg VØverhaug STrattner C(2024)Examining the merits of feature-specific similarity functions in the news domain using human judgmentsUser Modeling and User-Adapted Interaction10.1007/s11257-024-09412-234:4(995-1042)Online publication date: 7-Aug-2024
https://doi.org/10.1007/s11257-024-09412-2
Liu YAchananuparp PLim E(2024)Non-binary evaluation of next-basket food recommendationUser Modeling and User-Adapted Interaction10.1007/s11257-023-09369-834:1(183-227)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1007/s11257-023-09369-8
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten