Shaping the Future of Content-based News Recommenders: Insights from Evaluating Feature-Specific Similarity Metrics
Authors: 
Daniel Rosnes, 
Alain D. Starke, and 
Christoph TrattnerAuthors Info & Claims
Daniel Rosnes, Alain D. Starke, and Christoph TrattnerAuthors Info & ClaimsJune 2024
Pages 201 - 211
Abstract
In news media, recommender system technology faces several domain-specific challenges. The continuous stream of new content and users deems content-based recommendation strategies, based on similar-item retrieval, to remain popular. However, a persistent challenge is to select relevant features and corresponding similarity functions, and whether this depends on the specific context. We evaluated feature-specific similarity metrics using human similarity judgments across national and local news domains. We performed an online experiment (N = 141) where we asked participants to judge the similarity between pairs of randomly sampled news articles. We had three contributions: (1) comparing novel metrics based on large language models to ones traditionally used in news recommendations, (2) exploring differences in similarity judgments across national and local news domains, and (3) examining which content-based strategies were perceived as appropriate in the news domain. Our results showed that one of the novel large language model based metrics (SBERT) was highly correlated with human judgments, while there were only small, most non-significant differences across national and local news domains. Finally, we found that while it may be possible to automatically recommend similar news using feature-specific metrics, their representativeness and appropriateness varied. We explain how our findings can guide the design of future content-based and hybrid recommender strategies in the news domain.
References
[1]
Vimala Balakrishnan and Lloyd-Yemoh Ethel. 2014. Stemming and Lemmatization: A Comparison of Retrieval Performances. Lecture Notes on Software Engineering 2 (2014), 262–267. https://doi.org/10.7763/LNSE.2014.V2.134
[2]
Daniel Billsus and Michael J. Pazzani. 2000. User Modeling for Adaptive News Access. User Modeling and User-Adapted Interaction 10, 2 (2000), 147–180. https://doi.org/10.1023/A:1026501525781
[3]
Danushka Bollegala, Yutaka Matsuo, and Mitsuru Ishizuka. 2007. Measuring semantic similarity between words using Web search engines. In Proceedings of the 16th International Conference on World Wide Web (Banff, Alberta, Canada) (WWW ’07). Association for Computing Machinery, New York, NY, USA, 757–766. https://doi.org/10.1145/1242572.1242675
[4]
Benjamin P. Chamberlain, Emanuele Rossi, Dan Shiebler, Suvash Sedhain, and Michael M. Bronstein. 2020. Tuning Word2vec for Large Scale Recommendation Systems. In Proceedings of the 14th ACM Conference on Recommender Systems (Virtual Event, Brazil) (RecSys ’20). Association for Computing Machinery, New York, NY, USA, 732–737. https://doi.org/10.1145/3383313.3418486
[5]
Wei Chu, Seung-Taek Park, Todd Beaupre, Nitin Motgi, Amit Phadke, Seinjuti Chakraborty, and Joe Zachariah. 2009. A Case Study of Behavior-Driven Conjoint Analysis on Yahoo! Front Page Today Module. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (Paris, France) (KDD ’09). Association for Computing Machinery, New York, NY, USA, 1097–1104. https://doi.org/10.1145/1557019.1557138
[6]
Abhinandan S. Das, Mayur Datar, Ashutosh Garg, and Shyam Rajaram. 2007. Google News Personalization: Scalable Online Collaborative Filtering. In Proceedings of the 16th International Conference on World Wide Web (Banff, Alberta, Canada) (WWW ’07). Association for Computing Machinery, New York, NY, USA, 271–280. https://doi.org/10.1145/1242572.1242610
[7]
Mehdi Elahi, Dietmar Jannach, Lars Skjærven, Erik Knudsen, Helle Sjøvaag, Kristian Tolonen, Øyvind Holmstad, Igor Pipkin, Eivind Throndsen, Agnes Stenbom, Eivind Fiskerud, Adrian Oesch, Loek Vredenberg, and Christoph Trattner. 2022. Towards responsible media recommendation. AI and Ethics 2, 1 (2022), 103–114. https://doi.org/10.1007/s43681-021-00107-7
[8]
Florent Garcin, Boi Faltings, Olivier Donatsch, Ayar Alazzawi, Christophe Bruttin, and Amr Huber. 2014. Offline and Online Evaluation of News Recommender Systems at Swissinfo.Ch. In Proceedings of the 8th ACM Conference on Recommender Systems (Foster City, Silicon Valley, California, USA) (RecSys ’14). Association for Computing Machinery, New York, NY, USA, 169–176. https://doi.org/10.1145/2645710.2645745
[9]
Maarten Grootendorst. 2022. BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv preprint arXiv:2203.05794 (2022).
[10]
Hebatallah A. Mohamed Hassan, Giuseppe Sansonetti, Fabio Gasparetti, Alessandro Micarelli, and Joeran Beel. 2019. BERT, ELMo, USE and InferSent Sentence Encoders: The Panacea for Research-Paper Recommendation?. In ACM Conference on Recommender Systems.
[11]
Dietmar Jannach, Markus Zanker, Alexander Felfernig, and Gerhard Friedrich. 2010. Recommender Systems an Introduction. Cambridge University Press, Leiden. http://www.amazon.com/Recommender-Systems-Introduction-Dietmar-Jannach/dp/0521493366
[12]
Mozhgan Karimi, Dietmar Jannach, and Michael Jugovac. 2018. News recommender systems – Survey and roads ahead. Information Processing & Management 54, 6 (2018), 1203–1227. https://doi.org/10.1016/j.ipm.2018.04.008
[13]
Mohadeseh Kaviani and Hossein Rahmani. 2020. EmHash: Hashtag Recommendation using Neural Network based on BERT Embedding. In 2020 6th International Conference on Web Research (ICWR). 113–118. https://doi.org/10.1109/ICWR49608.2020.9122275
[14]
Peter Kolbeinsen Klingenberg. 2023. Using content-and behavioural data for recommendations in the Norwegian news market. Master’s thesis. The University of Bergen.
[15]
Erik Knudsen, Alain D. Starke, and Christoph Trattner. 2023. Topical Preference Trumps Other Features in News Recommendation: A Conjoint Analysis on a Representative Sample from Norway. In Proceedings of the International Workshop on News Recommendation and Analytics, co-located with the 2023 ACM Conference on Recommender Systems (RecSys 2023)(CEUR Workshop Proceedings, Vol. 3561), B. Kille (Ed.). CEUR-WS, Singapore, 14. https://hdl.handle.net/11245.1/7ef6ea51-8e5d-458c-bc0e-f134028fc912Singapore, 18 September 2023.
[16]
Per E Kummervold, Javier De la Rosa, Freddy Wetjen, and Svein Arne Brygfjeld. 2021. Operationalizing a National Digital Library: The Case for a Norwegian Transformer Model. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa). Linköping University Electronic Press, Sweden, Reykjavik, Iceland (Online), 20–29. https://aclanthology.org/2021.nodalida-main.3
[17]
Jingang Liu, Chunhe Xia, Xiaojian Li, Haihua Yan, and Tengteng Liu. 2020. A BERT-Based Ensemble Model for Chinese News Topic Prediction. In Proceedings of the 2020 2nd International Conference on Big Data Engineering (Shanghai, China) (BDE 2020). Association for Computing Machinery, New York, NY, USA, 18–23. https://doi.org/10.1145/3404512.3404524
[18]
Jingang Liu, Chunhe Xia, Xiaojian Li, Haihua Yan, and Tengteng Liu. 2020. A BERT-Based Ensemble Model for Chinese News Topic Prediction. In Proceedings of the 2020 2nd International Conference on Big Data Engineering (Shanghai, China) (BDE 2020). Association for Computing Machinery, New York, NY, USA, 18–23. https://doi.org/10.1145/3404512.3404524
[19]
Yuanhua Lv, Taesup Moon, Pranam Kolari, Zhaohui Zheng, Xuanhui Wang, and Yi Chang. 2011. Learning to Model Relatedness for News Recommendation. In Proceedings of the 20th International Conference on World Wide Web (Hyderabad, India) (WWW ’11). Association for Computing Machinery, New York, NY, USA, 57–66. https://doi.org/10.1145/1963405.1963417
[20]
Özlem Özgöbek, Jon Atle Gulla, and Riza Cenk Erdur. 2014. A Survey on Challenges and Methods in News Recommendation. In International Conference on Web Information Systems and Technologies.
[21]
Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Conference on Empirical Methods in Natural Language Processing.
[22]
Francesco Ricci, Lior Rokach, and Bracha Shapira. 2011. Introduction to Recommender Systems Handbook. Springer US, Boston, MA, 1–35. https://doi.org/10.1007/978-0-387-85820-3_1
[23]
Julio Rieis, Fabrício de Souza, Pedro Vaz de Melo, Raquel Prates, Haewoon Kwak, and Jisun An. 2021. Breaking the News: First Impressions Matter on Online News. Proceedings of the International AAAI Conference on Web and Social Media 9, 1 (Aug. 2021), 357–366. https://doi.org/10.1609/icwsm.v9i1.14619
[24]
Jose San Pedro and Stefan Siersdorfer. 2009. Ranking and classifying attractiveness of photos in folksonomies. 771–780. https://doi.org/10.1145/1526709.1526813
[25]
Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. arxiv:1409.1556 [cs.CV]
[26]
Helle Sjøvaag, Hallvard Moe, and Eirik Stavelin. 2012. Public service news on the Web: A large-scale content analysis of the Norwegian Broadcasting Corporation’s online news. Journalism Studies 13, 1 (2012), 90–106.
[27]
Vegard Rygh Solberg. 2022. News Recommendation based on Human Similarity Judgment. Master thesis. The University of Bergen. Masteroppgave i informasjonsvitenskap, INFO390, MASV-INFO.
[28]
A.D. Starke, Sebastian Øverhaug Larsen, and Christoph Trattner. 2021. Predicting Feature-based Similarity in the News Domain Using Human Judgments. In Proceedings of the 9th International Workshop on News Recommendation and Analytics (INRA 2021).
[29]
Nava Tintarev and Judith Masthoff. 2006. Similarity for News Recommender Systems. In Workshop on Recommender Systems and Intelligent User Interfaces, Gulden Uchyigit (Ed.). In conjunction with the International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, AH 2006, Dublin, Ireland, June 20-23, 2006.
[30]
Nava Tintarev and Judith Masthoff. 2006. Similarity for news recommender systems. In In Proceedings of the AH’06 Workshop on Recommender Systems and Intelligent User Interfaces. Citeseer.
[31]
Christoph Trattner and Dietmar Jannach. 2020. Learning to recommend similar items from human judgments. User Modeling and User-Adapted Interaction 30, 1 (2020), 1–49. https://doi.org/10.1007/s11257-019-09245-4
[32]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. arxiv:1706.03762 [cs.CL]
[33]
Amy Winecoff, Florin Brasoveanu, Bryce Casavant, Pearce Washabaugh, and Matthew Graham. 2019. Users in the Loop: A Psychologically-Informed Approach to Similar Item Retrieval.
[34]
Nakyeong Yang, Jeongje Jo, Myeongjun Jeon, Wooju Kim, and Juyoung Kang. 2022. Semantic and explainable research-related recommendation system based on semi-supervised methodology using BERT and LDA models. Expert Systems with Applications 190 (2022), 116209. https://doi.org/10.1016/j.eswa.2021.116209
[35]
Yuan Yao and F. Maxwell Harper. 2018. Judging Similarity: A User-Centric Study of Related Item Recommendations. In Proceedings of the 12th ACM Conference on Recommender Systems (Vancouver, British Columbia, Canada) (RecSys ’18). Association for Computing Machinery, New York, NY, USA, 288–296. https://doi.org/10.1145/3240323.3240351
[36]
Qi Zhang, Jingjie Li, Qinglin Jia, Chuyuan Wang, Jieming Zhu, Zhaowei Wang, and Xiuqiang He. 2021. UNBERT: User-News Matching BERT for News Recommendation. 3356–3362. https://doi.org/10.24963/ijcai.2021/462
[37]
Kun Zhou, Yuanhang Zhou, Wayne Xin Zhao, Xiaoke Wang, and Ji-Rong Wen. 2020. Towards Topic-Guided Conversational Recommender System. arxiv:2010.04125 [cs.CL]
Index Terms
- Shaping the Future of Content-based News Recommenders: Insights from Evaluating Feature-Specific Similarity Metrics
Recommendations
Judging similarity: a user-centric study of related item recommendations
RecSys '18: Proceedings of the 12th ACM Conference on Recommender SystemsRelated item recommenders operate in the context of a particular item. For instance, a music system's page about the artist Radio-head might recommend other similar artists such as The Flaming Lips. Often central to these recommendations is the ...
User-Specific Feature-Based Similarity Models for Top-n Recommendation of New Items
Survey Paper, Regular Papers and Special Section on Participatory Sensing and Crowd IntelligenceRecommending new items for suitable users is an important yet challenging problem due to the lack of preference history for the new items. Noncollaborative user modeling techniques that rely on the item features can be used to recommend new items. ...
An analysis of peer similarity for recommendations in P2P systems
In this paper, we propose a novel recommender framework for partially decentralized file sharing Peer-to-Peer systems. The proposed recommender system is based on user-based collaborative filtering. We take advantage from the partial search process used ...
Comments
Information & Contributors
Information
Published In
June 2024
338 pages
ISBN:9798400704338
DOI:10.1145/3627043
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 22 June 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- Research Council of Norway
Conference
UMAP '24: 32nd ACM Conference on User Modeling, Adaptation and Personalization
July 1 - 4, 2024
Cagliari, Italy
Acceptance Rates
Overall Acceptance Rate 162 of 633 submissions, 26%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 24Total Downloads
- Downloads (Last 12 months)24
- Downloads (Last 6 weeks)24
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatGet Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in