Just Accepted
Efficient Federated Recommender System with Adaptive Model Pruning and Momentum-based Batch Adjustment
Abstract
With the development of 5G communication and smart devices, the prosperity of online content has boosted the research of Recommender System (RS). Due to the data scarcity problem, researchers employ knowledge transfer techniques to improve the accuracy of RS. Data sharing or data augmentation are promising methods for such a problem, but the data suffers from privacy leakage during sharing. Thus, Federated Learning (FL) has been adopted to collaboratively train recommender models while preserving data privacy. Federated Recommender System (FRS) combines FL and RS to provide distributed recommendation services to the users. However, the existing FRS suffers from massive communication and system heterogeneity, where the necessity of model transmission and the diversity of the clients bring significant communication overhead to the system. In this paper, we propose Efficient Federated Recommender System with Adaptive Model Pruning and Momentum-based Batch Adjustment (eFRSA2) to reduce the communication overhead of FRS. eFRSA2 contains two modules. Adaptive Model Pruning utilizes magnitude pruning to reduce the communication volume and adaptively modifies the compression ratios of different clients to maintain the model accuracy. Momentum-based Batch Adjustment adjusts the local training batch number by a similar method of gradient descent with momentum to align the local computation time of the clients and reduce the communication overhead. The experimental results demonstrate that eFRSA2 can reduce up to 90% communication volume and mitigate the system heterogeneity by over 75%, demonstrating the priority of eFRSA2 in training efficiency. Source code can be found at https://github.com/shhjwu5/eFRSA2.
References
[1]
Durmus Alp Emre Acar, Yue Zhao, Ramon Matas Navarro, Matthew Mattina, Paul N. Whatmough, and Venkatesh Saligrama. 2021. Federated learning based on dynamic regularization. In Proceedings of the 9th International Conference on Learning Representations. OpenReview.net. https://openreview.net/forum?id=B7v4QMR6Z9w
[2]
Waqar Ali, Rajesh Kumar, Zhiyi Deng, Yansong Wang, and Jie Shao. 2021. A federated learning approach for privacy protection in context-aware recommender systems. Comput. J. 64, 7 (2021), 1016–1027. https://doi.org/10.1093/comjnl/bxab025
[3]
Muhammad Ammad-ud-din, Elena Ivannikova, Suleiman A. Khan, Were Oyomno, Qiang Fu, Kuan Eeik Tan, and Adrian Flanagan. 2019. Federated collaborative filtering for privacy-preserving personalized recommendation system. CoRR abs/1901.09888(2019). arXiv:1901.09888 http://arxiv.org/abs/1901.09888
[4]
Vito Walter Anelli, Yashar Deldjoo, Tommaso Di Noia, Antonio Ferrara, and Fedelucio Narducci. 2022. User-controlled federated matrix factorization for recommender systems. J. Intell. Inf. Syst. 58, 2 (2022), 287–309. https://doi.org/10.1007/s10844-021-00688-z
[5]
Javier Antorán, David Janz, James Urquhart Allingham, Erik A. Daxberger, Riccardo Barbano, Eric T. Nalisnick, and José Miguel Hernández-Lobato. 2022. Adapting the linearised Laplace model evidence for modern deep learning. In Proceedings of the 38th International Conference on Machine Learning. PMLR, 796–821. https://proceedings.mlr.press/v162/antoran22a.html
[6]
Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary. 2019. Federated learning with personalization layers. CoRR abs/1912.00818(2019). arXiv:1912.00818 http://arxiv.org/abs/1912.00818
[7]
Aayush Bansal, Rewa Sharma, and Mamta Kathuria. 2022. A systematic review on data scarcity problem in deep learning: Solution and applications. ACM Comput. Surv. 54, 10s (2022), 208:1–208:29. https://doi.org/10.1145/3502287
[8]
Jung-Hsien Chiang, Chung-Yao Ma, Chi-Shiang Wang, and Pei-Yi Hao. 2023. An adaptive, context-aware, and stacked attention network-based recommendation system to capture users’ temporal preference. IEEE Trans. Knowl. Data Eng. 35, 4 (2023), 3404–3418. https://doi.org/10.1109/TKDE.2022.3140387
[9]
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. 2021. Exploiting shared representations for personalized federated learning. In Proceedings of the 38th International Conference on Machine Learning. PMLR, 2089–2099. http://proceedings.mlr.press/v139/collins21a.html
[10]
Yingtong Dou, Zhiwei Liu, Li Sun, Yutong Deng, Hao Peng, and Philip S. Yu. 2020. Enhancing graph neural network-based fraud detectors against camouflaged fraudsters. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management. ACM, 315–324. https://doi.org/10.1145/3340531.3411903
[11]
Yongjie Du, Deyun Zhou, Yu Xie, Jiao Shi, and Maoguo Gong. 2021. Federated matrix factorization for privacy-preserving recommender systems. Appl. Soft Comput. 111(2021), 107700. https://www.sciencedirect.com/science/article/pii/S1568494621006219
[12]
Jibing Gong, Shen Wang, Jinlong Wang, Wenzheng Feng, Hao Peng, Jie Tang, and Philip S. Yu. 2020. Attentional graph convolutional networks for knowledge concept recommendation in MOOCs in a heterogeneous view. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 79–88. https://doi.org/10.1145/3397271.3401057
[13]
Hanxi Guo, Hao Wang, Tao Song, Yang Hua, Zhangcheng Lv, Xiulang Jin, Zhengui Xue, Ruhui Ma, and Haibing Guan. 2021. Siren: Byzantine-robust federated learning via proactive alarming. In Proceedings of the 12th ACM Symposium on Cloud Computing. ACM, 47–60. https://doi.org/10.1145/3472883.3486990
[14]
Lingbing Guo, Qiang Zhang, Zequn Sun, Mingyang Chen, Wei Hu, and Huajun Chen. 2022. Understanding and improving knowledge graph embedding for entity alignment. In Proceedings of the 38th International Conference on Machine Learning. PMLR, 8145–8156. https://proceedings.mlr.press/v162/guo22i.html
[15]
F. Maxwell Harper and Joseph A. Konstan. 2016. The MovieLens datasets: History and context. ACM Trans. Interact. Intell. Syst. 5, 4 (2016), 19:1–19:19. https://doi.org/10.1145/2827872
[16]
Yutao Huang, Lingyang Chu, Zirui Zhou, Lanjun Wang, Jiangchuan Liu, Jian Pei, and Yong Zhang. 2021. Personalized cross-silo federated learning on Non-IID data. In Proceedings of the 35th AAAI Conference on Artificial Intelligence. AAAI Press, 7865–7873. https://ojs.aaai.org/index.php/AAAI/article/view/16960
[17]
Mubashir Imran, Hongzhi Yin, Tong Chen, Quoc Viet Hung Nguyen, Alexander Zhou, and Kai Zheng. 2023. ReFRS: Resource-efficient federated recommender system for dynamic and diversified user preferences. ACM Trans. Inf. Syst. 41, 3 (2023), 65:1–65:30. https://doi.org/10.1145/3560486
[18]
Xiaopeng Jiang and Cristian Borcea. 2023. Complement sparsification: Low-overhead model pruning for federated learning. In Proceedings of the 37th AAAI Conference on Artificial Intelligence. AAAI Press, 8087–8095. https://ojs.aaai.org/index.php/AAAI/article/view/25977
[19]
Jie Li, Yongli Ren, and Ke Deng. 2022. FairGAN: GANs-based fairness-aware learning for recommendations with implicit feedback. In Proceedings of 2022 ACM Web Conference. ACM, 297–307. https://doi.org/10.1145/3485447.3511958
[20]
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. 2020. Federated optimization in heterogeneous networks. In Proceedings of Machine Learning and Systems. mlsys.org. https://proceedings.mlsys.org/book/316.pdf
[21]
Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou. 2021. FedBN: Federated learning on Non-IID features via local batch normalization. In Proceedings of the 9th International Conference on Learning Representations. OpenReview.net. https://openreview.net/forum?id=6YEQUn0QICG
[22]
Guanyu Lin, Feng Liang, Weike Pan, and Zhong Ming. 2021. FedRec: Federated Recommendation With Explicit Feedback. IEEE Intell. Syst. 36, 5 (2021), 21–30. https://doi.org/10.1109/MIS.2020.3017205
[23]
Yujie Lin, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Dongxiao Yu, Jun Ma, Maarten de Rijke, and Xiuzhen Cheng. 2020. Meta Matrix Factorization for Federated Rating Predictions. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval. ACM, 981–990. https://doi.org/10.1145/3397271.3401081
[24]
Shuchang Liu, Shuyuan Xu, Wenhui Yu, Zuohui Fu, Yongfeng Zhang, and Amélie Marian. 2021. FedCT: Federated collaborative transfer for recommendation. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 716–725. https://doi.org/10.1145/3404835.3462825
[25]
Zhiwei Liu, Yingtong Dou, Philip S. Yu, Yutong Deng, and Hao Peng. 2020. Alleviating the inconsistency problem of applying graph neural network to fraud detection. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 1569–1572. https://doi.org/10.1145/3397271.3401253
[26]
Zhiwei Liu, Liangwei Yang, Ziwei Fan, Hao Peng, and Philip S. Yu. 2022. Federated social recommendation with graph neural network. ACM Trans. Intell. Syst. Technol. 13, 4 (2022), 55:1–55:24. https://doi.org/10.1145/3501815
[27]
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics. PMLR, 1273–1282. http://proceedings.mlr.press/v54/mcmahan17a.html
[28]
Dheevatsa Mudigere, Yuchen Hao, Jianyu Huang, Zhihao Jia, Andrew Tulloch, Srinivas Sridharan, Xing Liu, Mustafa Ozdal, Jade Nie, Jongsoo Park, Liang Luo, Jie Amy Yang, Leon Gao, Dmytro Ivchenko, Aarti Basant, Yuxi Hu, Jiyan Yang, Ehsan K. Ardestani, Xiaodong Wang, Rakesh Komuravelli, Ching-Hsiang Chu, Serhat Yilmaz, Huayu Li, Jiyuan Qian, Zhuobo Feng, Yinbin Ma, Junjie Yang, Ellie Wen, Hong Li, Lin Yang, Chonglin Sun, Whitney Zhao, Dimitry Melts, Krishna Dhulipala, K. R. Kishore, Tyler Graf, Assaf Eisenman, Kiran Kumar Matam, Adi Gangidi, Guoqiang Jerry Chen, Manoj Krishnan, Avinash Nayak, Krishnakumar Nair, Bharath Muthiah, Mahmoud khorashadi, Pallab Bhattacharya, Petr Lapukhov, Maxim Naumov, Ajit Mathews, Lin Qiao, Mikhail Smelyanskiy, Bill Jia, and Vijay Rao. 2022. Software-hardware co-design for fast and scalable training of deep learning recommendation models. In Proceedings of the 49th Annual International Symposium on Computer Architecture. ACM, 993–1011. https://doi.org/10.1145/3470496.3533727
[29]
Khalil Muhammad, Qinqin Wang, Diarmuid O’Reilly-Morgan, Elias Z. Tragos, Barry Smyth, Neil Hurley, James Geraci, and Aonghus Lawlor. 2020. FedFast: Going beyond average for faster training of federated recommender systems. In Proceedings of the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. ACM, 1234–1242. https://doi.org/10.1145/3394486.3403176
[30]
Hao Peng, Jianxin Li, Yu He, Yaopeng Liu, Mengjiao Bao, Lihong Wang, Yangqiu Song, and Qiang Yang. 2018. Large-scale hierarchical text classification with recursively regularized deep Graph-CNN. In Proceedings of the 2018 World Wide Web Conference on World Wide Web. ACM, 1063–1072. https://doi.org/10.1145/3178876.3186005
[31]
Vasileios Perifanis and Pavlos S. Efraimidis. 2022. Federated neural collaborative filtering. Knowl. Based Syst. 242(2022), 108441. https://doi.org/10.1016%2Fj.knosys.2022.108441
[32]
Geet Sethi, Bilge Acun, Niket Agarwal, Christos Kozyrakis, Caroline Trippel, and Carole-Jean Wu. 2022. RecShard: Statistical feature-based memory optimization for industry-scale neural recommendation. In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems. ACM, 344–358. https://doi.org/10.1145/3503222.3507777
[33]
Hongjian Shi, Ruhui Ma, Dongmei Li, and Haibing Guan. 2023. Hierarchical adaptive collaborative learning: A distributed learning framework for customized cloud services in 6G mobile systems. IEEE Netw. 37, 2 (2023), 44–53. https://doi.org/10.1109/MNET.001.2200382
[34]
Hongjian Shi, Weichu Zheng, Zifei Liu, Ruhui Ma, and Haibing Guan. 2023. Automatic pipeline parallelism: A parallel inference framework for deep learning applications in 6G mobile communication systems. IEEE J. Sel. Areas Commun. 41, 7 (2023), 2041–2056. https://doi.org/10.1109/JSAC.2023.3280970
[35]
Qinyong Wang, Hongzhi Yin, Tong Chen, Junliang Yu, Alexander Zhou, and Xiangliang Zhang. 2022. Fast-adapting and privacy-preserving federated recommender system. VLDB J. 31, 5 (2022), 877–896. https://doi.org/10.1007/s00778-021-00700-6
[36]
Lianghao Xia, Chao Huang, Yong Xu, Jiashu Zhao, Dawei Yin, and Jimmy X. Huang. 2022. Hypergraph contrastive collaborative filtering. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 70–79. https://doi.org/10.1145/3477495.3532058
[37]
Liu Yang, Ben Tan, Vincent W. Zheng, Kai Chen, and Qiang Yang. 2020. Federated recommendation systems. In Federated Learning - Privacy and Incentive. Lecture Notes in Computer Science, Vol. 12500. Springer, 225–239. https://doi.org/10.1007/978-3-030-63076-8_16
[38]
Zhen Yang, Ming Ding, Bin Xu, Hongxia Yang, and Jie Tang. 2022. STAM: A spatiotemporal aggregation method for graph neural network-based recommendation. In Proceedings of 2022 ACM Web Conference. ACM, 3217–3228. https://doi.org/10.1145/3485447.3512041
[39]
Jianqing Zhang, Yang Hua, Hao Wang, Tao Song, Zhengui Xue, Ruhui Ma, and Haibing Guan. 2023. FedALA: Adaptive local aggregation for personalized federated learning. In Proceedings of the 37th AAAI Conference on Artificial Intelligence. AAAI Press, 11237–11244. https://ojs.aaai.org/index.php/AAAI/article/view/26330
[40]
Mengqi Zhang, Shu Wu, Xueli Yu, Qiang Liu, and Liang Wang. 2023. Dynamic graph neural networks for sequential recommendation. IEEE Trans. Knowl. Data Eng. 35, 5 (2023), 4741–4753. https://doi.org/10.1109/TKDE.2022.3151618
[41]
Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, Hao Peng, Jianxin Li, Jia Wu, Ziwei Liu, Pengtao Xie, Caiming Xiong, Jian Pei, Philip S. Yu, and Lichao Sun. 2023. A comprehensive survey on pretrained foundation models: A history from BERT to ChatGPT. CoRR abs/2302.09419(2023). arXiv:2302.09419 https://doi.org/10.48550/arXiv.2302.09419
[42]
Kun Zhou, Hui Yu, Wayne Xin Zhao, and Ji-Rong Wen. 2022. Filter-enhanced MLP is all you need for sequential recommendation. In Proceedings of the ACM Web Conference 2022. ACM, 2388–2399. https://doi.org/10.1145/3485447.3512111
[43]
Pan Zhou, Kehao Wang, Linke Guo, Shimin Gong, and Bolong Zheng. 2021. A privacy-preserving distributed contextual federated online learning framework with big data support in social recommender systems. IEEE Trans. Knowl. Data Eng. 33, 3 (2021), 824–838. https://doi.org/10.1109/TKDE.2019.2936565
[44]
Zhuangdi Zhu, Junyuan Hong, and Jiayu Zhou. 2021. Data-free knowledge distillation for heterogeneous federated learning. In Proceedings of the 38th International Conference on Machine Learning. PMLR, 12878–12889. http://proceedings.mlr.press/v139/zhu21b.html
Index Terms
- Efficient Federated Recommender System with Adaptive Model Pruning and Momentum-based Batch Adjustment
Recommendations
Improving Accuracy of Recommender System by Item Clustering
Recommender System (RS) predicts user's ratings towards items, and then recommends highly-predicted items to user. In recent years, RS has been playing more and more important role in the agent research field. There have been a great deal of researches ...
A New Approach for Recommender System
ICACS '17: Proceedings of the 1st International Conference on Algorithms, Computing and SystemsIn today's e-commerce environment, Collaborative Filtering (CF) is a widely used algorithm for recommender system, which is to identify the users who have similar preferences to the target user, and to predict the preference of the target user according ...
Combining Memory-Based and Model-Based Collaborative Filtering in Recommender System
PACCS '09: Proceedings of the 2009 Pacific-Asia Conference on Circuits, Communications and SystemsCollaborative filtering (CF) technique has been proved to be one of the most successful techniques in recommender systems. Two types of algorithms for collaborative filtering have been researched: memory-based CF and model-based CF. Memory-based ...
Comments
Information & Contributors
Information
Published In
EISSN:2770-6699
Table of ContentsCopyright © 2024 Copyright held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Online AM: 13 November 2024
Accepted: 29 October 2024
Revised: 23 September 2024
Received: 15 August 2023
Check for updates
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 135Total Downloads
- Downloads (Last 12 months)135
- Downloads (Last 6 weeks)47
Reflects downloads up to 20 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in