research-article

Critically Examining the Claimed Value of Convolutions over User-Item Embedding Maps for Recommender Systems

Authors:

Maurizio Ferrari Dacrema,

Federico Parroni,

Paolo Cremonesi,

Dietmar JannachAuthors Info & Claims

CIKM '20: Proceedings of the 29th ACM International Conference on Information & Knowledge Management

Pages 355 - 363

https://doi.org/10.1145/3340531.3411901

Published: 19 October 2020 Publication History

Abstract

In recent years, algorithm research in the area of recommender systems has shifted from matrix factorization techniques and their latent factor models to neural approaches. However, given the proven power of latent factor models, some newer neural approaches incorporate them within more complex network architectures. One specific idea, recently put forward by several researchers, is to consider potential correlations between the latent factors, i.e., embeddings, by applying convolutions over the user-item interaction map. However, contrary to what is claimed in these articles, such interaction maps do not share the properties of images where Convolutional Neural Networks (CNNs) are particularly useful. In this work, we show through analytical considerations and empirical evaluations that the claimed gains reported in the literature cannot be attributed to the ability of CNNs to model embedding correlations, as argued in the original papers. Moreover, additional performance evaluations show that all of the examined recent CNN-based models are outperformed by existing non-neural machine learning techniques or traditional nearest-neighbor approaches. On a more general level, our work points to major methodological issues in recommender systems research.

Supplementary Material

MP4 File (3340531.3411901.mp4)

Presentation Video

Download
13.11 MB

References

[1]

Sebastiano Antenucci, Simone Boglio, Emanuele Chioso, Ervin Dervishaj, Shuwen Kang, Tommaso Scarlatti, and Maurizio Ferrari Dacrema. 2018. Artist-driven Layering and User's Behaviour Impact on Recommendations in a Playlist Continuation Scenario. In Recommender Systems Challenge Workshop at the 12th ACM Conference on Recommender Systems. 4:1--4:6.

Digital Library

[2]

Timothy G. Armstrong, Alistair Moffat, William Webber, and Justin Zobel. 2009. Improvements That Don't Add Up: Ad-hoc Retrieval Results Since 1998. In Proceedings of CIKM '09. 601--610.

Digital Library

[3]

Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. 2018. An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling. arXiv:1803.01271.

[4]

James Bennett and Stan Lanning. 2007. The Netflix Prize. In KDD Cup Workshop 2007.

[5]

Colin Cooper, Sang Hyuk Lee, Tomasz Radzik, and Yiannis Siantos. 2014. Random walks in recommender systems: exact computation and simulations. In Proceedings of WWW '14. 811--816.

Digital Library

[6]

Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations. In Proceedings of RecSys '16. 191--198.

Digital Library

[7]

Paolo Cremonesi, Yehuda Koren, and Roberto Turrin. 2010. Performance of Recommender Algorithms on Top-n Recommendation Tasks. In Proceedings of RecSys '10. 39--46.

Digital Library

[8]

Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional neural networks on graphs with fast localized spectral filtering. In Proceedings of NIPS '16. 3844--3852.

[9]

Xiaoyu Du, Xiangnan He, Fajie Yuan, Jinhui Tang, Zhiguang Qin, and Tat-Seng Chua. 2019. Modeling Embedding Dimension Correlations via Convolutional Neural Collaborative Filtering. ACM TOIS, Vol. 37, 4.

Digital Library

[10]

Maurizio Ferrari Dacrema, Simone Boglio, Paolo Cremonesi, and Dietmar Jannach. 2019 a. A Troubling Analysis of Reproducibility and Progress in Recommender Systems Research. arXiv:1911.07698.

[11]

Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. 2019 b. Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches. Proceedings of RecSys '19.

Digital Library

[12]

Antonino Freno, Martin Saveski, Rodolphe Jenatton, and Cédric Archambeau. 2015. One-Pass Ranking Models for Low-Latency Product Recommendations. In Proceedings of KDD '15. 1789--1798. http://doi.acm.org/10.1145/2783258.2788579

Digital Library

[13]

Xiangnan He, Xiaoyu Du, Xiang Wang, Feng Tian, Jinhui Tang, and Tat-Seng Chua. 2018. Outer Product-based Neural Collaborative Filtering. In Proceedings of IJCAI '18. 2227--2233.

[14]

Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural collaborative filtering. In Proceedings of WWW '17. 173--182.

Digital Library

[15]

José Miguel Hernández-Lobato, Matthew W Hoffman, and Zoubin Ghahramani. 2014. Predictive Entropy Search for Efficient Global Optimization of Black-box Functions. In Proceedings of NIPS '14. 918--926.

[16]

Yifan Hu, Yehuda Koren, and Chris Volinsky. 2008. Collaborative Filtering for Implicit Feedback Datasets. In Proceedings of ICDM '08. 263--272.

Digital Library

[17]

Junyang Jiang, Deqing Yang, Yanghua Xiao, and Chenlu Shen. 2019. Convolutional Gaussian Embeddings for Personalized Recommendation with Uncertainty. In Proceedings of IJCAI '19. 2642--2648.

[18]

Christopher C Johnson. 2014. Logistic matrix factorization for implicit feedback data. In Proceedings of NIPS '14, Vol. 27.

[19]

Yehuda Koren and Robert Bell. 2011. Advances in Collaborative Filtering. 145--186.

[20]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Proceedings of NIPS '12. 1097--1105.

[21]

Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. 1998. Gradient-based learning applied to document recognition. Proc. IEEE, 2278--2324.

[22]

Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y Ng. 2009. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In Proceedings of ICML '09. 609--616.

Digital Library

[23]

Mark Levy and Kris Jack. 2013. Efficient top-n recommendation by linear regression. In ACM RecSys 2013 Large Scale Recommender Systems Workshop.

[24]

Dawen Liang, Rahul G Krishnan, Matthew D Hoffman, and Tony Jebara. 2018. Variational Autoencoders for Collaborative Filtering. In Proceedings of WWW '18. 689--698.

Digital Library

[25]

Jimmy Lin. 2019. The Neural Hype, Justified! A Recantation. SIGIR Forum, Vol. 53, 2 (2019), 88--93.

Digital Library

[26]

Zachary C. Lipton and Jacob Steinhardt. 2018. Troubling Trends in Machine Learning Scholarship. In Proceedings of ICML '18: The Debates .arxiv: 1807.03341

[27]

Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully convolutional networks for semantic segmentation. In Proceedings of CVPR '13. 3431--3440.

[28]

Malte Ludewig and Dietmar Jannach. 2018. Evaluation of Session-based Recommendation Algorithms. UMUAI, Vol. 28, 4--5, 331--390.

Digital Library

[29]

Malte Ludewig, Noemi Mauro, Sara Latifi, and Dietmar Jannach. 2019. Performance Comparison of Neural and Non-Neural Approaches to Session-based Recommendation. In Proceedings of RecSys '19.

Digital Library

[30]

Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. 2018. Statistical and Machine Learning forecasting methods: Concerns and ways forward. PloS one, Vol. 13. Issue 3.

[31]

Xia Ning and George Karypis. 2011. SLIM: Sparse linear methods for top-n recommender systems. In Proceedings of ICDM '11. 497--506.

Digital Library

[32]

Bibek Paudel, Fabian Christoffel, Chris Newell, and Abraham Bernstein. 2017. Updatable, Accurate, Diverse, and Scalable Recommendations for Interactive Applications. ACM TiiS, Vol. 7, 1.

Digital Library

[33]

Steffen Rendle. 2010. Factorization Machines. In Proceedings of ICDM '10. 995--1000.

[34]

Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009. BPR: Bayesian personalized ranking from implicit feedback. In UAI '09. 452--461.

Digital Library

[35]

Steffen Rendle, Walid Krichene, Li Zhang, and John Anderson. 2020. Neural Collaborative Filtering vs. Matrix Factorization Revisited. In Proceedings of RecSys '20.

Digital Library

[36]

Steffen Rendle, Li Zhang, and Yehuda Koren. 2019. On the Difficulty of Evaluating Baselines: A Study on Recommender Systems. CoRR, Vol. abs/1905.01395. http://arxiv.org/abs/1905.01395

[37]

Paul Resnick, Neophytos Iacovou, Mitesh Suchak, Peter Bergstrom, and John Riedl. 1994. GroupLens: An Open Architecture for Collaborative Filtering of Netnews. In Proceedings of CSCW '94. 175--186.

Digital Library

[38]

Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. 2017. Dynamic routing between capsules. In Proceedings of NIPS '17. 3856--3866.

[39]

Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. 2001. Item-based Collaborative Filtering Recommendation Algorithms. In Proceedings of WWW '01. 285--295.

Digital Library

[40]

Jiaxi Tang and Ke Wang. 2018. Personalized top-n sequential recommendation via convolutional sequence embedding. In Proceedings of WSDM '18. 565--573.

Digital Library

[41]

Srinivas C Turaga, Joseph F Murray, Viren Jain, Fabian Roth, Moritz Helmstaedter, Kevin Briggman, Winfried Denk, and H Sebastian Seung. 2010. Convolutional networks can learn to generate affinity graphs for image segmentation. Neural Computation, Vol. 22, 511--538. Issue 2.

Digital Library

[42]

Aaron Van den Oord, Sander Dieleman, and Benjamin Schrauwen. 2013. Deep content-based music recommendation. In Proceedings of NIPS '13. 2643--2651.

[43]

Xin Xin, Bo Chen, Xiangnan He, Dong Wang, Yue Ding, and Joemon Jose. 2019. CFM: Convolutional Factorization Machines for Context-Aware Recommendation. In Proceedings of IJCAI '19. 3926--3932.

[44]

Wei Yang, Kuang Lu, Peilin Yang, and Jimmy Lin. 2019. Critically Examining the Neural Hype: Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models. In Proceedings of SIGIR '19. 1129--1132.

Digital Library

[45]

Fisher Yu and Vladlen Koltun. 2015. Multi-Scale Context Aggregation by Dilated Convolutions. CoRR, Vol. abs/1511.07122. https://arxiv.org/abs/1511.07122

[46]

Fajie Yuan, Alexandros Karatzoglou, Ioannis Arapakis, Joemon M. Jose, and Xiangnan He. 2019. A Simple Convolutional Generative Network for Next Item Recommendation. In Proceedings of WSDM '19. 582--590.

Digital Library

[47]

Bangzuo Zhang, Haobo Zhang, Xiaoxin Sun, Guozhong Feng, and Chunguang He. 2018b. Integrating an Attention Mechanism and Convolution Collaborative Filtering for Document Context-Aware Rating Prediction. IEEE Access, Vol. 7, 3826--3835.

[48]

Quangui Zhang, Longbing Cao, Chengzhang Zhu, Zhiqiang Li, and Jinguang Sun. 2018a. CoupledCF: Learning Explicit and Implicit User-item Couplings in Recommendation for Deep Collaborative Filtering. In Proceedings of IJCAI '18. 3662--3668.

[49]

Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. 2019. Deep Learning Based Recommender System: A Survey and New Perspectives. Comput. Surveys, Article 5, 5:1--5:38 pages.

Cited By

Ge YLiu SFu ZTan JLi ZXu SLi YXian YZhang Y(2024)A Survey on Trustworthy Recommender SystemsACM Transactions on Recommender Systems10.1145/3652891Online publication date: 13-Apr-2024
https://doi.org/10.1145/3652891
Luo ZHuang ZTang JHou YGao Y(2024)Time-aware Self-Attention Meets Logic Reasoning in Recommender Systems2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10651142(1-8)Online publication date: 30-Jun-2024
https://doi.org/10.1109/IJCNN60899.2024.10651142
Cai XXia LRen XHuang CFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)How Expressive are Graph Neural Networks in Recommendation?Proceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3614917(173-182)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3614917
Show More Cited By

Index Terms

Critically Examining the Claimed Value of Convolutions over User-Item Embedding Maps for Recommender Systems
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Neural networks
2. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Recommender systems

Recommendations

Collaborative Variational Autoencoder for Recommender Systems
KDD '17: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Modern recommender systems usually employ collaborative filtering with rating information to recommend items to users due to its successful performance. However, because of the drawbacks of collaborative-based methods such as sparsity, cold start, etc., ...
Attacking item-based recommender systems with power items
RecSys '14: Proceedings of the 8th ACM Conference on Recommender systems

Recommender Systems (RS) are vulnerable to attack by malicious users who intend to bias the recommendations for their own benefit. Research in this area has developed attack models, detection methods, and mitigation schemes to understand and protect ...
Acquiring User Information Needs for Recommender Systems
WI-IAT '13: Proceedings of the 2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT) - Volume 03

Most recommender systems attempt to use collaborative filtering, content-based filtering or hybrid approach to recommend items to new users. Collaborative filtering recommends items to new users based on their similar neighbours, and content-based ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CIKM '20: Proceedings of the 29th ACM International Conference on Information & Knowledge Management

October 2020

3619 pages

ISBN:9781450368599

DOI:10.1145/3340531

General Chairs:
Mathieu d'Aquin
DSI, Insight, NUI Galway, Ireland
,
Stefan Dietze
GESIS, Cologne, Germany, Heinrich-Heine-University Düsseldorf, Germany, L3S Research Center, Germany
,
Program Chairs:
Claudia Hauff
TU Delft, The Netherlands
,
Edward Curry
DSI, Insight, NUI Galway, Ireland
,
Philippe Cudre Mauroux
eXascale, University of Fribourg, Switzerland

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CIKM '20

Sponsor:

CIKM '20: The 29th ACM International Conference on Information and Knowledge Management

October 19 - 23, 2020

Virtual Event, Ireland

Acceptance Rates

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
278
Total Downloads

Downloads (Last 12 months)24
Downloads (Last 6 weeks)6

Reflects downloads up to 16 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ge YLiu SFu ZTan JLi ZXu SLi YXian YZhang Y(2024)A Survey on Trustworthy Recommender SystemsACM Transactions on Recommender Systems10.1145/3652891Online publication date: 13-Apr-2024
https://doi.org/10.1145/3652891
Luo ZHuang ZTang JHou YGao Y(2024)Time-aware Self-Attention Meets Logic Reasoning in Recommender Systems2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10651142(1-8)Online publication date: 30-Jun-2024
https://doi.org/10.1109/IJCNN60899.2024.10651142
Cai XXia LRen XHuang CFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)How Expressive are Graph Neural Networks in Recommendation?Proceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3614917(173-182)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3614917
Chen CMa WZhang MWang CLiu YMa S(2023)Revisiting Negative Sampling vs. Non-sampling in Implicit RecommendationACM Transactions on Information Systems10.1145/352267241:1(1-25)Online publication date: 25-Feb-2023
https://dl.acm.org/doi/10.1145/3522672
Rashed AElsayed SSchmidt-Thieme L(2022)Context and Attribute-Aware Sequential Recommendation via Cross-AttentionProceedings of the 16th ACM Conference on Recommender Systems10.1145/3523227.3546777(71-80)Online publication date: 12-Sep-2022
https://dl.acm.org/doi/10.1145/3523227.3546777
Zhang YYin GDong HZhang L(2022)Attention-based Frequency-aware Multi-scale Network for Sequential RecommendationApplied Soft Computing10.1016/j.asoc.2022.109349127:COnline publication date: 1-Sep-2022
https://dl.acm.org/doi/10.1016/j.asoc.2022.109349
Pérez Maurera FFerrari Dacrema MCremonesi P(2022)An Evaluation Study of Generative Adversarial Networks for Collaborative FilteringAdvances in Information Retrieval10.1007/978-3-030-99736-6_45(671-685)Online publication date: 10-Apr-2022
https://dl.acm.org/doi/10.1007/978-3-030-99736-6_45
Kim JWi JKim Y(2021)Sequential Recommendations on GitHub RepositoryApplied Sciences10.3390/app1104158511:4(1585)Online publication date: 10-Feb-2021
https://doi.org/10.3390/app11041585
Cremonesi PJannach D(2021)Progress in recommender systems researchAI Magazine10.1609/aimag.v42i3.1814542:3(43-54)Online publication date: 1-Sep-2021
https://dl.acm.org/doi/10.1609/aimag.v42i3.18145
Anelli VBellogín ADi Noia TPomo C(2021)Reenvisioning the comparison between Neural Collaborative Filtering and Matrix FactorizationProceedings of the 15th ACM Conference on Recommender Systems10.1145/3460231.3475944(521-529)Online publication date: 13-Sep-2021
https://dl.acm.org/doi/10.1145/3460231.3475944
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents