Cited By
View all- Pu YFan RChen JZhu ZLian DChen E(2024)Automated Sparse and Low-Rank Shallow Autoencoders for RecommendationACM Transactions on Recommender Systems10.1145/3656482Online publication date: 10-Apr-2024
AutoS2AE: Automate to Regularize Sparse Shallow Autoencoders for Recommendation
Authors: 
Rui Fan, Yuanhao Pu, Jin Chen, Zhihao Zhu, Defu Lian, 
Enhong ChenAuthors Info & Claims
Rui Fan, Yuanhao Pu, Jin Chen, Zhihao Zhu, Defu Lian, Enhong ChenAuthors Info & ClaimsPages 1032 - 1042
Abstract
The Embarrassingly Shallow Autoencoders (EASE and SLIM) are strong recommendation methods based on implicit feedback, compared to competing methods like iALS and VAE-CF. However, EASE suffers from several major shortcomings. First, the training and inference of EASE can not scale with the increasing number of items since it requires storing and inverting a large dense matrix; Second, though its optimization objective – the square loss– can yield a closed-form solution, it is not consistent with recommendation goal – predicting a personalized ranking on a set of items, so that its performance is far from optimal w.r.t ranking-oriented recommendation metrics. Finally, the regularization coefficients are sensitive w.r.t recommendation accuracy and vary a lot across different datasets, so the fine-tuning of these parameters is important yet time-consuming. To improve training and inference efficiency, we propose a Similarity-Structure Aware Shallow Autoencoder on top of three similarity structures, including Co-Occurrence, KNN and NSW. We then optimize the model with a weighted square loss, which is proven effective for ranking-based recommendation but still capable of deriving closed-form solutions. However, the weight in the loss can not be learned in the training set and is similarly sensitive w.r.t the accuracy to regularization coefficients. To automatically tune the hyperparameters, we design two validation losses on the validation set for guidance, and update the hyperparameters with the gradient of the validation losses. We finally evaluate the proposed method on multiple real-world datasets and show that it outperforms seven competing baselines remarkably, and verify the effectiveness of each part in the proposed method.
References
[1]
Shun-ichi Amari. 1993. Backpropagation and stochastic gradient descent method. Neurocomputing 5, 4-5 (1993), 185–196.
[2]
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450 (2016).
[3]
Immanuel Bayer, Xiangnan He, Bhargav Kanagal, and Steffen Rendle. 2017. A generic coordinate descent framework for learning from implicit feedback. In Proceedings of the 26th International Conference on World Wide Web. 1341–1350.
[4]
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. 2015. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349 (2015).
[5]
Jin Chen, Defu Lian, Binbin Jin, Xu Huang, Kai Zheng, and Enhong Chen. 2022. Fast variational autoencoder with inverted multi-index for collaborative filtering. In Proceedings of the ACM Web Conference 2022. 1944–1954.
[6]
Yihong Chen, Bei Chen, Xiangnan He, Chen Gao, Yong Li, Jian-Guang Lou, and Yue Wang. 2019. λ opt: Learn to regularize recommender models in finer levels. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 978–986.
[7]
Yifan Chen and Maarten de Rijke. 2018. A collective variational autoencoder for top-n recommendation with side information. In Proceedings of the 3rd workshop on deep learning for recommender systems. 3–9.
[8]
Evangelia Christakopoulou and George Karypis. 2016. Local item-item models for top-n recommendation. In Proceedings of the 10th ACM Conference on Recommender Systems. 67–74.
[9]
Chao Feng, Wuchao Li, Defu Lian, Zheng Liu, and Enhong Chen. 2022. Recommender Forest for Efficient Retrieval. Advances in Neural Information Processing Systems 35 (2022), 38912–38924.
[10]
Chao Feng, Defu Lian, Xiting Wang, Zheng Liu, Xing Xie, and Enhong Chen. 2023. Reinforcement Routing on Proximity Graph for Efficient Recommendation. ACM Transactions on Information Systems 41, 1 (2023), 1–27.
[11]
Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. 2019. Are we really making much progress¿ A worrying analysis of recent neural recommendation approaches. In Proceedings of the 13th ACM conference on recommender systems. 101–109.
[12]
Daniel Gabay and Bertrand Mercier. 1976. A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Computers & mathematics with applications 2, 1 (1976), 17–40.
[13]
Yifan Hu, Yehuda Koren, and Chris Volinsky. 2008. Collaborative filtering for implicit feedback datasets. In 2008 Eighth IEEE international conference on data mining. Ieee, 263–272.
[14]
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4700–4708.
[15]
Binbin Jin, Defu Lian, Zheng Liu, Qi Liu, Jianhui Ma, Xing Xie, and Enhong Chen. 2020. Sampling-decomposable generative adversarial recommender. Advances in Neural Information Processing Systems 33 (2020), 22629–22639.
[16]
Santosh Kabbur, Xia Ning, and George Karypis. 2013. Fism: factored item similarity models for top-n recommender systems. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining. 659–667.
[17]
Daeryong Kim and Bongwon Suh. 2019. Enhancing VAEs for collaborative filtering: flexible priors & gating mechanisms. In Proceedings of the 13th ACM Conference on Recommender Systems. 403–407.
[18]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[19]
Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013).
[20]
Xiaopeng Li and James She. 2017. Collaborative variational autoencoder for recommender systems. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining. 305–314.
[21]
Defu Lian, Qi Liu, and Enhong Chen. 2020. Personalized ranking with importance sampling. In Proceedings of The Web Conference 2020. 1093–1103.
[22]
Defu Lian, Haoyu Wang, Zheng Liu, Jianxun Lian, Enhong Chen, and Xing Xie. 2020. Lightrec: A memory and search-efficient recommender system. In Proceedings of The Web Conference 2020. 695–705.
[23]
Defu Lian, Yongji Wu, Yong Ge, Xing Xie, and Enhong Chen. 2020. Geography-aware sequential location recommendation. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. 2009–2019.
[24]
Dawen Liang, Rahul G Krishnan, Matthew D Hoffman, and Tony Jebara. 2018. Variational autoencoders for collaborative filtering. In Proceedings of the 2018 world wide web conference. 689–698.
[25]
Dawen Liang, Rahul G Krishnan, Matthew D Hoffman, and Tony Jebara. 2018. Variational autoencoders for collaborative filtering. In Proceedings of the 2018 world wide web conference. 689–698.
[26]
Yu A Malkov and Dmitry A Yashunin. 2018. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE transactions on pattern analysis and machine intelligence 42, 4 (2018), 824–836.
[27]
Paolo Massa and Paolo Avesani. 2004. Trust-aware collaborative filtering for recommender systems. In OTM Confederated International Conferences" On the Move to Meaningful Internet Systems". Springer, 492–508.
[28]
Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In Icml.
[29]
Xia Ning and George Karypis. 2011. Slim: Sparse linear methods for top-n recommender systems. In 2011 IEEE 11th international conference on data mining. IEEE, 497–506.
[30]
Xia Ning and George Karypis. 2012. Sparse linear methods with side information for top-n recommendations. In Proceedings of the sixth ACM conference on Recommender systems. 155–162.
[31]
Rong Pan and Martin Scholz. 2009. Mind the gaps: weighting the unknown in large-scale one-class collaborative filtering. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining. 667–676.
[32]
Rong Pan, Yunhong Zhou, Bin Cao, Nathan N Liu, Rajan Lukose, Martin Scholz, and Qiang Yang. 2008. One-class collaborative filtering. In 2008 Eighth IEEE International Conference on Data Mining. IEEE, 502–511.
[33]
Prajit Ramachandran, Barret Zoph, and Quoc V Le. 2017. Searching for activation functions. arXiv preprint arXiv:1710.05941 (2017).
[34]
Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2012. BPR: Bayesian personalized ranking from implicit feedback. arXiv preprint arXiv:1205.2618 (2012).
[35]
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning. PMLR, 1278–1286.
[36]
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning. PMLR, 1278–1286.
[37]
Martin Riedmiller and Heinrich Braun. 1993. A direct adaptive method for faster backpropagation learning: The RPROP algorithm. In IEEE international conference on neural networks. IEEE, 586–591.
[38]
Suvash Sedhain, Aditya Krishna Menon, Scott Sanner, and Lexing Xie. 2015. Autorec: Autoencoders meet collaborative filtering. In Proceedings of the 24th international conference on World Wide Web. 111–112.
[39]
Ilya Shenbin, Anton Alekseev, Elena Tutubalina, Valentin Malykh, and Sergey I Nikolenko. 2020. Recvae: A new variational autoencoder for top-n recommendations with implicit feedback. In Proceedings of the 13th International Conference on Web Search and Data Mining. 528–536.
[40]
Ankur Sinha, Pekka Malo, and Kalyanmoy Deb. 2017. A review on bilevel optimization: from classical to evolutionary approaches and applications. IEEE Transactions on Evolutionary Computation 22, 2 (2017), 276–295.
[41]
Harald Steck. 2019. Embarrassingly shallow autoencoders for sparse data. In The World Wide Web Conference. 3251–3257.
[42]
Harald Steck. 2020. Autoencoders that don’t overfit towards the identity. Advances in Neural Information Processing Systems 33 (2020), 19598–19608.
[43]
Harald Steck, Maria Dimakopoulou, Nickolai Riabov, and Tony Jebara. 2020. Admm slim: Sparse recommendations for many users. In Proceedings of the 13th International Conference on Web Search and Data Mining. 555–563.
[44]
Tim Van Erven and Peter Harremos. 2014. Rényi divergence and Kullback-Leibler divergence. IEEE Transactions on Information Theory 60, 7 (2014), 3797–3820.
[45]
Yao Wu, Christopher DuBois, Alice X Zheng, and Martin Ester. 2016. Collaborative denoising auto-encoders for top-n recommender systems. In Proceedings of the ninth ACM international conference on web search and data mining. 153–162.
[46]
Yongji Wu, Defu Lian, Neil Zhenqiang Gong, Lu Yin, Mingyang Yin, Jingren Zhou, and Hongxia Yang. 2021. Linear-time self attention with codeword histogram for efficient recommendation. In Proceedings of the Web Conference 2021. 1262–1273.
Index Terms
- AutoS2AE: Automate to Regularize Sparse Shallow Autoencoders for Recommendation
Recommendations
Automated Sparse and Low-Rank Shallow Autoencoders for Recommendation
Collaborative filtering (CF) works have demonstrated the robust capabilities of Shallow Autoencoders on implicit feedback, showcasing highly competitive performance with other reasonable approaches (e.g., iALS and VAE-CF). However, despite their dual ...
A Collaborative Filtering Recommendation Algorithm Based on Item Classification
PACCS '09: Proceedings of the 2009 Pacific-Asia Conference on Circuits, Communications and SystemsCollaborative filtering systems represent services of personalized that aim at predicting a user’s interest on some items available in the application systems. With the development of electronic commerce, the number of users and items grows rapidly, ...
Learning Item/User Vectors from Comments for Collaborative Recommendation
ICMLC '17: Proceedings of the 9th International Conference on Machine Learning and ComputingCollaborative Filtering (CF) has been widely used in many recommender systems over the past decades. Conventional CF-based methods mainly consider the ratings given to items via users and suffer from the sparsity and cold-start problems very much. ...
Comments
Information & Contributors
Information
Published In
April 2023
4293 pages
ISBN:9781450394161
DOI:10.1145/3543507
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 30 April 2023
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
Conference
WWW '23
Sponsor:
Acceptance Rates
Overall Acceptance Rate 1,899 of 8,196 submissions, 23%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 277Total Downloads
- Downloads (Last 12 months)152
- Downloads (Last 6 weeks)3
Reflects downloads up to 30 Aug 2024
Other Metrics
Citations
Cited By
View all- Pu YFan RChen JZhu ZLian DChen E(2024)Automated Sparse and Low-Rank Shallow Autoencoders for RecommendationACM Transactions on Recommender Systems10.1145/3656482Online publication date: 10-Apr-2024
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format