Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article

Fast-adapting and privacy-preserving federated recommender system

Published: 03 October 2021 Publication History

Abstract

In the mobile Internet era, recommender systems have become an irreplaceable tool to help users discover useful items, thus alleviating the information overload problem. Recent research on deep neural network (DNN)-based recommender systems have made significant progress in improving prediction accuracy, largely attributed to the widely accessible large-scale user data. Such data is commonly collected from users’ personal devices and then centrally stored in the cloud server to facilitate model training. However, with the rising public concerns on user privacy leakage in online platforms, online users are becoming increasingly anxious over abuses of user privacy. Therefore, it is urgent and beneficial to develop a recommender system that can achieve both high prediction accuracy and strong privacy protection. To this end, we propose a DNN-based recommendation model called PrivRec running on the decentralized federated learning (FL) environment, which ensures that a user’s data is fully retained on her/his personal device while contributing to training an accurate model. On the other hand, to better embrace the data heterogeneity (e.g., users’ data vary in scale and quality significantly) in FL, we innovatively introduce a first-order meta-learning method that enables fast on-device personalization with only a few data points. Furthermore, to defend against potential malicious participants that pose serious security threat to other users, we further develop a user-level differentially private model, namely DP-PrivRec, so attackers are unable to identify any arbitrary user from the trained model. To compensate for the loss by adding noise during model updates, we introduce a two-stage training approach. Finally, we conduct extensive experiments on two large-scale datasets in a simulated FL environment, and the results validate the superiority of both PrivRec and DP-PrivRec.

References

[1]
Abadi, M., Chu, A., Goodfellow, I., McMahan, H.B., Mironov, I., Talwar, K., Zhang, L.: Deep learning with differential privacy. In: SIGSAC, pp. 308–318 (2016)
[2]
Acar A, Aksu H, Uluagac AS, and Conti M A survey on homomorphic encryption schemes: theory and implementation ACM Comput. Surv. (CSUR) 2018 51 4 1-35
[3]
Ammad-Ud-Din, M., Ivannikova, E., Khan, S.A., Oyomno, W., Fu, Q., Tan, K.E., Flanagan, A.: Federated collaborative filtering for privacy-preserving personalized recommendation system. arXiv:1901.09888 (2019)
[4]
Bonawitz, K., Eichner, H., Grieskamp, W., Huba, D., Ingerman, A., Ivanov, V., Kiddon, C., Konečnỳ, J., Mazzocchi, S., McMahan, H.B., et al.: Towards federated learning at scale: System design. arXiv:1902.01046 (2019)
[5]
Bonawitz, K., Ivanov, V., Kreuter, B., Marcedone, A., McMahan, H.B., Patel, S., Ramage, D., Segal, A., Seth, K.: Practical secure aggregation for privacy-preserving machine learning. In: ACM SIGSAC Conference on Computer and Communications Security, pp. 1175–1191 (2017)
[6]
Briggs, C., Fan, Z., Andras, P.: Federated learning with hierarchical clustering of local updates to improve training on non-iid data. In: IJCNN, pp. 1–9. IEEE (2020)
[7]
Caron, M., Bojanowski, P., Joulin, A., Douze, M.: Deep clustering for unsupervised learning of visual features. In: ECCV, pp. 132–149 (2018)
[8]
Caruana R Multitask learning. Mach. Learn. 1997 28 1 41-75
[9]
Chen, F., Dong, Z., Li, Z., He, X.: Federated meta-learning for recommendation. arXiv:1802.07876 (2018)
[10]
Chen, T., Yin, H., Chen, H., Yan, R., Nguyen, Q.V.H., Li, X.: Air: Attentional intention-aware recommender systems. In: ICDE, pp. 304–315. IEEE (2019)
[11]
Chen, T., Yin, H., Zheng, Y., Huang, Z., Wang, Y., Wang, M.: Learning elastic embeddings for customizing on-device recommenders. KDD (2021)
[12]
Cremonesi, P., Koren, Y., Turrin, R.: Performance of recommender algorithms on top-n recommendation tasks. In: RecSys, pp. 39–46 (2010)
[13]
Deng, Y., Kamani, M.M., Mahdavi, M.: Adaptive personalized federated learning. arXiv:2003.13461 (2020)
[14]
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805 (2018)
[15]
Doersch, C., Gupta, A., Efros, A.A.: Unsupervised visual representation learning by context prediction. In: ICCV, pp. 1422–1430 (2015)
[16]
Dwork, C.: Differential privacy: A survey of results. In: International conference on theory and applications of models of computation, pp. 1–19. Springer (2008)
[17]
Dwork, C., Kenthapadi, K., McSherry, F., Mironov, I., Naor, M.: Our data, ourselves: Privacy via distributed noise generation. In: Annual International Conference on the Theory and Applications of Cryptographic Techniques, pp. 486–503 (2006)
[18]
Dwork, C., McSherry, F., Nissim, K., Smith, A.: Calibrating noise to sensitivity in private data analysis. In: Theory of cryptography conference, pp. 265–284. Springer (2006)
[19]
Dwork C, Roth A, et al. The algorithmic foundations of differential privacy Found. Trends Theor. Comput. Sci. 2014 9 3–4 211-407
[20]
Evans D, Kolesnikov V, and Rosulek M A pragmatic introduction to secure multi-party computation Found. Trends Privacy Secur. 2017 2 2-3
[21]
Fallah, A., Mokhtari, A., Ozdaglar, A.: Personalized federated learning: a meta-learning approach. arXiv:2002.07948 (2020)
[22]
Finn, C., Abbeel, P., Levine, S.: Model-agnostic meta-learning for fast adaptation of deep networks. In: ICML, pp. 1126–1135 (2017)
[23]
Flanagan, A., Oyomno, W., Grigorievskiy, A., Tan, K.E., Khan, S.A., Ammad-Ud-Din, M.: Federated multi-view matrix factorization for personalized recommendations. arXiv:2004.04256 (2020)
[24]
Fung BC, Wang K, Chen R, and Yu PS Privacy-preserving data publishing: a survey of recent developments ACM Comput. Surv. 2010 42 4 1-53
[25]
Geyer, R.C., Klein, T., Nabi, M.: Differentially private federated learning: A client level perspective. arXiv:1712.07557 (2017)
[26]
Guo, L., Yin, H., Wang, Q., Chen, T., Zhou, A., Quoc Viet Hung, N.: Streaming session-based recommendation. In: KDD, pp. 1569–1577 (2019)
[27]
Hamm, J., Cao, Y., Belkin, M.: Learning privately from multiparty data. In: ICML, pp. 555–563 (2016)
[28]
Hard, A., Rao, K., Mathews, R., Ramaswamy, S., Beaufays, F., Augenstein, S., Eichner, H., Kiddon, C., Ramage, D.: Federated learning for mobile keyboard prediction. rXiv:1811.03604 (2018)
[29]
Hardy, S., Henecka, W., Ivey-Law, H., Nock, R., Patrini, G., Smith, G., Thorne, B.: Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption. arXiv:1711.10677 (2017)
[30]
He, X., Liao, L., Zhang, H., Nie, L., Hu, X., Chua, T.S.: Neural collaborative filtering. In: WWW, pp. 173–182 (2017)
[31]
Hidasi, B., Karatzoglou, A., Baltrunas, L., Tikk, D.: Session-based recommendations with recurrent neural networks. arXiv:1511.06939 (2015)
[32]
Hidasi, B., Quadrana, M., Karatzoglou, A., Tikk, D.: Parallel recurrent neural network architectures for feature-rich session-based recommendations. In: RecSys, pp. 241–248 (2016)
[33]
Huang, P.S., He, X., Gao, J., Deng, L., Acero, A., Heck, L.: Learning deep structured semantic models for web search using clickthrough data. In: CIKM, pp. 2333–2338 (2013)
[34]
Hung NQV, Viet HH, Tam NT, Weidlich M, Yin H, and Zhou X Computing crowd consensus with partial agreement TKDE 2017 30 1 1-14
[35]
Jalalirad, A., Scavuzzo, M., Capota, C., Sprague, M.: A simple and efficient federated recommender system. In: IEEE/ACM International Conference on Big Data Computing, Applications and Technologies, pp. 53–58 (2019)
[36]
Jayaraman, B., Evans, D.: Evaluating differentially private machine learning in practice. In: USENIX, pp. 1895–1912 (2019)
[37]
Jiang, Y., Konečnỳ, J., Rush, K., Kannan, S.: Improving federated learning personalization via model agnostic meta learning. arXiv:1909.12488 (2019)
[38]
Kasiviswanathan SP, Lee HK, Nissim K, Raskhodnikova S, and Smith A What can we learn privately? SIAM J. Comput. 2011 40 3 793-826
[39]
Khodak, M., Balcan, M.F.F., Talwalkar, A.S.: Adaptive gradient-based meta-learning methods. In: Neurips, pp. 5917–5928 (2019)
[40]
Kolesnikov, A., Zhai, X., Beyer, L.: Revisiting self-supervised visual representation learning. In: CVPR, pp. 1920–1929 (2019)
[41]
Kong, L., d’Autume, C.d.M., Ling, W., Yu, L., Dai, Z., Yogatama, D.: A mutual information maximization perspective of language representation learning. arXiv:1910.08350 (2019)
[42]
Le, Q., Mikolov, T.: Distributed representations of sentences and documents. In: ICML, pp. 1188–1196. PMLR (2014)
[43]
Li, T., Song, L., Fragouli, C.: Federated recommendation system via differential privacy. arXiv:2005.06670 (2020)
[44]
Ma, C., Li, J., Ding, M., Yang, H.H., Shu, F., Quek, T.Q., Poor, H.V.: On safeguarding privacy and security in the framework of federated learning. IEEE Network (2020)
[45]
Makri, E., Rotaru, D., Smart, N.P., Vercauteren, F.: Epic: efficient private image classification (or: Learning from the masters). In: Cryptographers Track at the RSA Conference, pp. 473–492. Springer (2019)
[46]
Mansour, Y., Mohri, M., Ro, J., Suresh, A.T.: Three approaches for personalization with applications to federated learning. arXiv:2002.10619 (2020)
[47]
Masi, I., Wu, Y., Hassner, T., Natarajan, P.: Deep face recognition: A survey. In: SIBGRAPI, pp. 471–478. IEEE (2018)
[48]
McMahan, H.B., Andrew, G., Erlingsson, U., Chien, S., Mironov, I., Papernot, N., Kairouz, P.: A general approach to adding differential privacy to iterative training procedures. arXiv:1812.06210 (2018)
[49]
McMahan, H.B., Ramage, D., Talwar, K., Zhang, L.: Learning differentially private recurrent language models. arXiv:1710.06963 (2017)
[50]
Melis, L., Song, C., De Cristofaro, E., Shmatikov, V.: Exploiting unintended feature leakage in collaborative learning. In: IEEE Symposium on Security and Privacy, pp. 691–706 (2019)
[51]
Nasr, M., Shokri, R., Houmansadr, A.: Comprehensive privacy analysis of deep learning: Stand-alone and federated learning under passive and active white-box inference attacks. arXiv:1812.00910 (2018)
[52]
Nguyen QVH, Duong CT, Nguyen TT, Weidlich M, Aberer K, Yin H, and Zhou X Argument discovery via crowdsourcing VLDB J. 2017 26 4 511-535
[53]
Nichol, A., Achiam, J., Schulman, J.: On first-order meta-learning algorithms. arXiv:1803.02999 (2018)
[54]
Nock, R., Hardy, S., Henecka, W., Ivey-Law, H., Patrini, G., Smith, G., Thorne, B.: Entity resolution and federated learning get a federated resolution. arXiv:1803.04035 (2018)
[55]
Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv:1807.03748 (2018)
[56]
Orekondy, T., Oh, S.J., Zhang, Y., Schiele, B., Fritz, M.: Gradient-leaks: Understanding and controlling deanonymization in federated learning. arXiv:1805.05838 (2018)
[57]
Phan, N., Wu, X., Hu, H., Dou, D.: Adaptive laplace mechanism: Differential privacy preservation in deep learning. In: ICDM, pp. 385–394. IEEE (2017)
[58]
Qiu R, Huang Z, Li J, and Yin H Exploiting cross-session information for session-based recommendation with graph neural networks TOIS 2020 38 3 1-23
[59]
Rahman MA, Rahman T, Laganière R, Mohammed N, and Wang Y Membership inference attack against differentially private deep learning model Trans. Data Privacy 2018 11 1 61-79
[60]
Rajeswaran, A., Finn, C., Kakade, S., Levine, S.: Meta-learning with implicit gradients (2019)
[61]
Ramaswamy, S., Mathews, R., Rao, K., Beaufays, F.: Federated learning for emoji prediction in a mobile keyboard. arXiv:1906.04329 (2019)
[62]
Ribero, M., Henderson, J., Williamson, S., Vikalo, H.: Federating recommendations using differentially private prototypes. arXiv:2003.00602 (2020)
[63]
Ryffel, T., Trask, A., Dahl, M., Wagner, B., Mancuso, J., Rueckert, D., Passerat-Palmbach, J.: A generic framework for privacy preserving deep learning. rXiv:1811.04017 (2018)
[64]
Smith V, Chiang CK, Sanjabi M, and Talwalkar AS Federated multi-task learning. Neurips 2017 30 4424-4434
[65]
Song, S., Chaudhuri, K., Sarwate, A.D.: Stochastic gradient descent with differentially private updates. In: IEEE Global Conference on Signal and Information Processing, pp. 245–248 (2013)
[66]
Sun K, Qian T, Chen T, Liang Y, Nguyen QVH, and Yin H Where to go next: modeling long-and short-term user preferences for point-of-interest recommendation AAAI 2020 34 214-221
[67]
Thakkar, O., Andrew, G., McMahan, H.B.: Differentially private learning with adaptive clipping. arXiv:1905.03871 (2019)
[68]
Truex, S., Liu, L., Gursoy, M.E., Wei, W., Yu, L.: Effects of differential privacy and data skewness on membership inference vulnerability. In: 2019 First IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA), pp. 82–91. IEEE (2019)
[69]
Uhlerop C, Slavković A, and Fienberg SE Privacy-preserving data sharing for genome-wide association studies J. Privacy Confidentiality 2013 5 1 137
[70]
Vanschoren, J.: Meta-learning: A survey. arXiv:1810.03548 (2018)
[71]
Vilalta R and Drissi Y A perspective view and survey of meta-learning Artif. Intell. Rev. 2002 18 2 77-95
[72]
Wang, H., Kaplan, Z., Niu, D., Li, B.: Optimizing federated learning on non-iid data with reinforcement learning. In: INFOCOM, pp. 1698–1707. IEEE (2020)
[73]
Wang, M., Ren, P., Mei, L., Chen, Z., Ma, J., de Rijke, M.: A collaborative session-based recommendation approach with parallel memory modules. In: SIGIR, pp. 345–354 (2019)
[74]
Wang, Q., Yin, H., Chen, T., Huang, Z., Wang, H., Zhao, Y., Viet Hung, N.Q.: Next point-of-interest recommendation on resource-constrained mobile devices. In: TheWebConf, pp. 906–916 (2020)
[75]
Wang, Q., Yin, H., Hu, Z., Lian, D., Wang, H., Huang, Z.: Neural memory streaming recommender networks with adversarial training. In: KDD, pp. 2467–2475 (2018)
[76]
Wang, S., Cao, L., Wang, Y., Sheng, Q.Z., Orgun, M., Lian, D.: A survey on session-based recommender systems. arXiv:1902.04864 (2019)
[77]
Wang S, Tuor T, Salonidis T, Leung KK, Makaya C, He T, and Chan K Adaptive federated learning in resource constrained edge computing systems IEEE J. Sel. Areas Commun. 2019 37 6 1205-1221
[78]
Wang, Y., Yin, H., Chen, H., Wo, T., Xu, J., Zheng, K.: Origin-destination matrix prediction via graph convolution: a new perspective of passenger demand modeling. In: KDD, pp. 1227–1235 (2019)
[79]
Wang, Y.X., Balle, B., Kasiviswanathan, S.P.: Subsampled rényi differential privacy and analytical moments accountant. In: International Conference on Artificial Intelligence and Statistics, pp. 1226–1235 (2019)
[80]
Wang, Z., Song, M., Zhang, Z., Song, Y., Wang, Q., Qi, H.: Beyond inferring class representatives: User-level privacy leakage from federated learning. In: INFOCOM, pp. 2512–2520. IEEE (2019)
[81]
Wang, Z., Wei, W., Cong, G., Li, X.L., Mao, X.L., Qiu, M.: Global context enhanced graph neural networks for session-based recommendation. In: SIGIR, pp. 169–178 (2020)
[82]
Wu, C., Wu, F., Cao, Y., Huang, Y., Xie, X.: Fedgnn: Federated graph neural network for privacy-preserving recommendation. KDD (2021)
[83]
Xia, X., Yin, H., Yu, J., Wang, Q., Cui, L., Zhang, X.: Self-supervised hypergraph convolutional networks for session-based recommendation. arXiv:2012.06852 (2020)
[84]
Xu J, Glicksberg BS, Su C, Walker P, Bian J, and Wang F Federated learning for healthcare informatics J. Healthcare Inform. Res. 2021 5 1 1-19
[85]
Yang Q, Liu Y, Chen T, and Tong Y Federated machine learning: concept and applications TIST 2019 10 2 1-19
[86]
Yin, H., Wang, Q., Zheng, K., Li, Z., Yang, J., Zhou, X.: Social influence-based group representation learning for group recommendation. In: ICDE, pp. 566–577 (2019)
[87]
Zhang, J., Wang, J., Zhao, Y., Chen, B.: An efficient federated learning scheme with differential privacy in mobile edge computing. In: International Conference on Machine Learning and Intelligent Communications, pp. 538–550. Springer (2019)
[88]
Zhang S, Yao L, Sun A, and Tay Y Deep learning based recommender system: a survey and new perspectives ACM Comput. Surv. 2019 52 1 1-38
[89]
Zhang, S., Yin, H., Chen, T., Huang, Z., Cui, L., Zhang, X.: Graph embedding for recommendation against attribute inference attacks. In: WWW, pp. 3002–3014 (2021)
[90]
Zhang, S., Yin, H., Chen, T., Hung, Q.V.N., Huang, Z., Cui, L.: Gcn-based user representation learning for unifying robust recommendation and fraudster detection. In: SIGIR, pp. 689–698 (2020)
[91]
Zhang, Y., Yang, Q.: A survey on multi-task learning. arXiv:1707.08114 (2017)
[92]
Zhang, Y., Yin, H., Huang, Z., Du, X., Yang, G., Lian, D.: Discrete deep learning for fast content-aware recommendation. In: WSDM, pp. 717–726 (2018)
[93]
Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., Chandra, V.: Federated learning with non-iid data. arXiv:1806.00582 (2018)

Cited By

View all
  • (2024)Privacy-preserving Cross-domain Recommendation with Federated Graph LearningACM Transactions on Information Systems10.1145/365344842:5(1-29)Online publication date: 13-May-2024
  • (2024)Decentralized Collaborative Learning with Adaptive Reference Data for On-Device POI RecommendationProceedings of the ACM Web Conference 202410.1145/3589334.3645696(3930-3939)Online publication date: 13-May-2024
  • (2024)Towards Personalized Privacy: User-Governed Data Contribution for Federated RecommendationProceedings of the ACM Web Conference 202410.1145/3589334.3645690(3910-3918)Online publication date: 13-May-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image The VLDB Journal — The International Journal on Very Large Data Bases
The VLDB Journal — The International Journal on Very Large Data Bases  Volume 31, Issue 5
Sep 2022
339 pages

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 03 October 2021
Accepted: 31 August 2021
Revision received: 08 July 2021
Received: 07 January 2021

Author Tags

  1. Recommender system
  2. Federated learning
  3. Meta-learning

Qualifiers

  • Research-article

Funding Sources

  • Australian Research Council

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)0
  • Downloads (Last 6 weeks)0
Reflects downloads up to 12 Sep 2024

Other Metrics

Citations

Cited By

View all
  • (2024)Privacy-preserving Cross-domain Recommendation with Federated Graph LearningACM Transactions on Information Systems10.1145/365344842:5(1-29)Online publication date: 13-May-2024
  • (2024)Decentralized Collaborative Learning with Adaptive Reference Data for On-Device POI RecommendationProceedings of the ACM Web Conference 202410.1145/3589334.3645696(3930-3939)Online publication date: 13-May-2024
  • (2024)Towards Personalized Privacy: User-Governed Data Contribution for Federated RecommendationProceedings of the ACM Web Conference 202410.1145/3589334.3645690(3910-3918)Online publication date: 13-May-2024
  • (2024)Prompt-enhanced Federated Content Representation Learning for Cross-domain RecommendationProceedings of the ACM Web Conference 202410.1145/3589334.3645337(3139-3149)Online publication date: 13-May-2024
  • (2024)Parallel Fractional Stochastic Gradient Descent With Adaptive Learning for Recommender SystemsIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2022.318521235:3(470-483)Online publication date: 1-Mar-2024
  • (2024)Heterogeneous Workload-Based Consumer Resource Recommendation Model for Smart Cities: eHealth Edge–Cloud Connectivity Using Federated Split LearningIEEE Transactions on Consumer Electronics10.1109/TCE.2024.337446270:1(4187-4196)Online publication date: 19-Mar-2024
  • (2023)Manipulating Visually Aware Federated Recommender Systems and Its CountermeasuresACM Transactions on Information Systems10.1145/363000542:3(1-26)Online publication date: 23-Oct-2023
  • (2023)Trustworthy Recommendation and Search: Introduction to the Special Section - Part 2ACM Transactions on Information Systems10.1145/360477641:4(1-6)Online publication date: 28-Jul-2023
  • (2023)Trustworthy Recommendation and Search: Introduction to the Special Issue - Part 1ACM Transactions on Information Systems10.1145/357999541:3(1-5)Online publication date: 7-Feb-2023
  • (2023)Poisoning GNN-based Recommender Systems with Generative Surrogate-based AttacksACM Transactions on Information Systems10.1145/356742041:3(1-24)Online publication date: 7-Feb-2023
  • Show More Cited By

View Options

View options

Get Access

Login options

Full Access

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media