research-article

Open access

Cache-Enabled Federated Learning Systems

Authors:

Carlee Joe-Wong,

Stratis Ioannidis,

Marie SiewAuthors Info & Claims

MobiHoc '23: Proceedings of the Twenty-fourth International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing

Pages 1 - 11

https://doi.org/10.1145/3565287.3610264

Published: 16 October 2023 Publication History

Abstract

Federated learning (FL) is a distributed paradigm for collaboratively learning models without having clients disclose their private data. One natural and practically relevant metric to measure the efficiency of FL algorithms is the total wall-clock training time, which can be quantified by the product of the average time needed for a single iteration and the number of iterations for convergence. In this work, we focus on improving FL efficiency with respect to this metric through caching. Specifically, instead of having all clients download the latest global model from a parameter server, we select a subset of clients to access, with a smaller delay, a somewhat stale global model stored in caches. We propose CacheFL - a cache-enabled variant of FedAvg, and provide theoretical convergence guarantees in the general setting where the local data is imbalanced and heterogeneous. Armed with this result, we determine the caching strategies that minimize total wall-clock training time at a given convergence threshold for both stochastic and deterministic communication/computation delays. Through numerical experiments on real data traces, we show the advantage of our proposed scheme against several baselines, over both synthetic and real-world datasets.

Supplementary Material

PDF File (p1-liu-supp.pdf)

Supplemental material.

Download
1.50 MB

References

[1]

Duane Buck and Mukesh Singhal. 1996. An analytic study of caching in computer systems. J. Parallel and Distrib. Comput. 32, 2 (1996), 205--214.

Digital Library

[2]

Chen Chen, Hong Xu, Wei Wang, Baochun Li, Bo Li, Li Chen, and Gong Zhang. 2021. Communication-efficient federated learning with adaptive parameter freezing. In 2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS). IEEE, 1--11.

[3]

Mingzhe Chen, H Vincent Poor, Walid Saad, and Shuguang Cui. 2020. Convergence time optimization for federated learning over wireless networks. IEEE Transactions on Wireless Communications 20, 4 (2020), 2457--2471.

[4]

Mingzhe Chen, Zhaohui Yang, Walid Saad, Changchuan Yin, H Vincent Poor, and Shuguang Cui. 2020. A joint learning and communications framework for federated learning over wireless networks. IEEE Transactions on Wireless Communications 20, 1 (2020), 269--283.

Digital Library

[5]

Alp Emre Durmus, Zhao Yue, Matas Ramon, Mattina Matthew, Whatmough Paul, and Saligrama Venkatesh. 2021. Federated Learning Based on Dynamic Regularization. In International Conference on Learning Representations.

[6]

Fangcheng Fu, Xupeng Miao, Jiawei Jiang, Huanran Xue, and Bin Cui. 2022. Towards communication-efficient vertical federated learning training via cache-enabled local updates. arXiv preprint arXiv:2207.14628 (2022).

[7]

Google. 2022. How Messages improves suggestions with federated technology. https://support.google.com/messages/answer/9327902 Last accessed 22 July 2022.

[8]

Eduard Gorbunov, Filip Hanzely, and Peter Richtárik. 2021. Local sgd: Unified theory and new efficient methods. In International Conference on Artificial Intelligence and Statistics. PMLR, 3556--3564.

[9]

Xinran Gu, Kaixuan Huang, Jingzhao Zhang, and Longbo Huang. 2021. Fast federated learning in the presence of arbitrary device unavailability. Advances in Neural Information Processing Systems 34 (2021), 12052--12064.

[10]

Farzin Haddadpour, Mohammad Mahdi Kamani, Aryan Mokhtari, and Mehrdad Mahdavi. 2021. Federated learning with compression: Unified analysis and sharp guarantees. In International Conference on Artificial Intelligence and Statistics. PMLR, 2350--2358.

[11]

Kais Hamza, Peter Jagers, Aidan Sudbury, and Daniel Tokarev. 2009. The mixing advantage is less than 2. Extremes 12, 1 (2009), 19--31.

[12]

Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. 2018. Federated learning for mobile keyboard prediction. arXiv preprint arXiv:1811.03604 (2018).

[13]

Gautam Kamath. 2015. Bounds on the expectation of the maximum of samples from a gaussian. (2015). www.gautamkamath.com/writings/gaussian_max.pdf

[14]

Mehmet Karaca, Tansu Alpcan, and Ozgur Ercetin. 2019. Smart Scheduling and Feedback Allocation over Non-stationary Wireless Channels. arXiv preprint arXiv:1911.03632 (2019).

[15]

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. 2020. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning. PMLR, 5132--5143.

[16]

Jakub Konečnỳ, H Brendan McMahan, Daniel Ramage, and Peter Richtárik. 2016. Federated optimization: Distributed machine learning for on-device intelligence. arXiv preprint arXiv:1610.02527 (2016).

[17]

Alex Krizhevsky, Geoffrey Hinton, et al. 2009. Learning multiple layers of features from tiny images. (2009).

[18]

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE (1998), 2278--2324.

[19]

Dongsheng Li, Yuxi Zhao, and Xiaowen Gong. 2021. Quality-Aware distributed computation and communication scheduling for fast convergent wireless federated learning. In 2021 19th International Symposium on Modeling and Optimization in Mobile, Ad hoc, and Wireless Networks (WiOpt). IEEE, 1--8.

[20]

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. 2020. Federated optimization in heterogeneous networks. Proceedings of Machine Learning and Systems 2 (2020), 429--450.

[21]

Yuezhou Liu. 2023. Appendix for Cache-enabled Federated Learning Systems. Technical Report. Available as http://www1.ece.neu.edu/~liuyuezhou/Cache_enabled_FL_TP_Yuezhou.pdf.

[22]

Yuezhou Liu, Yuanyuan Li, Lili Su, Edmund Yeh, and Stratis Ioannidis. 2022. Experimental design networks: A paradigm for serving heterogeneous learners under networking constraints. In IEEE INFOCOM 2022-IEEE Conference on Computer Communications. IEEE, 210--219.

Digital Library

[23]

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. 2017. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics. PMLR.

[24]

Mobiperf. [n. d.]. https://www.measurementlab.net/tests/mobiperf/.

[25]

Georgios Paschos, George Iosifidis, Giuseppe Caire, et al. 2020. Cache optimization models and algorithms. Foundations and Trends® in Communications and Information Theory 16, 3--4 (2020), 156--345.

[26]

Dario Rossi and Giuseppe Rossini. 2011. Caching performance of content centric networks under multi-path routing (and more). Relatório técnico, Telecom ParisTech 2011 (2011), 1--6.

[27]

Yichen Ruan, Xiaoxi Zhang, and Carlee Joe-Wong. 2021. How valuable is your data? optimizing client recruitment in federated learning. In 2021 19th International Symposium on Modeling and Optimization in Mobile, Ad hoc, and Wireless Networks (WiOpt). IEEE, 1--8.

[28]

Yichen Ruan, Xiaoxi Zhang, Shu-Che Liang, and Carlee Joe-Wong. 2021. Towards flexible device participation in federated learning. In International Conference on Artificial Intelligence and Statistics. PMLR, 3403--3411.

[29]

Zai Shi and Atilla Eryilmaz. 2021. Communication-efficient Subspace Methods for High-dimensional Federated Learning. In 2021 17th International Conference on Mobility, Sensing and Networking (MSN). IEEE, 543--550.

[30]

Sebastian U Stich and Sai Praneeth Karimireddy. 2020. The error-feedback framework: Better rates for SGD with delayed gradients and compressed updates. Journal of Machine Learning Research 21 (2020), 1--36.

[31]

Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan McMahan, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, et al. 2021. A field guide to federated optimization. arXiv preprint arXiv:2107.06917 (2021).

[32]

Su Wang, Yichen Ruan, Yuwei Tu, Satyavrat Wagle, Christopher G Brinton, and Carlee Joe-Wong. 2021. Network-aware optimization of distributed learning for fog computing. IEEE/ACM Transactions on Networking 29, 5 (2021), 2019--2032.

Digital Library

[33]

Blake Woodworth, Kumar Kshitij Patel, Sebastian Stich, Zhen Dai, Brian Bullins, Brendan Mcmahan, Ohad Shamir, and Nathan Srebro. 2020. Is local SGD better than minibatch SGD?. In International Conference on Machine Learning. PMLR.

[34]

Yuting Wu, Yanxiang Jiang, Mehdi Bennis, Fuchun Zheng, Xiqi Gao, and Xiaohu You. 2020. Content popularity prediction in fog radio access networks: A federated learning based approach. In ICC 2020-2020 IEEE International Conference on Communications (ICC). IEEE, 1--6.

[35]

Han Xiao, Kashif Rasul, and Roland Vollgraf. 2017. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017).

[36]

Cong Xie, Sanmi Koyejo, and Indranil Gupta. 2019. Asynchronous federated optimization. arXiv preprint arXiv:1903.03934 (2019).

[37]

Chenhao Xu, Youyang Qu, Yong Xiang, and Longxiang Gao. 2021. Asynchronous federated learning on heterogeneous devices: A survey. arXiv preprint arXiv:2109.04269 (2021).

[38]

Hang Xu, Chen-Yu Ho, Ahmed M Abdelmoniem, Aritra Dutta, El Houcine Bergou, Konstantinos Karatsenidis, Marco Canini, and Panos Kalnis. 2020. Compressed communication for distributed deep learning: Survey and quantitative evaluation. Technical Report.

[39]

Haibo Yang, Jia Liu, and Elizabeth S Bentley. 2021. CFedAvg: achieving efficient communication and fast convergence in non-iid federated learning. In 2021 19th International Symposium on Modeling and Optimization in Mobile, Ad hoc, and Wireless Networks (WiOpt). IEEE, 1--8.

[40]

Shuai Yu, Xu Chen, Zhi Zhou, Xiaowen Gong, and Di Wu. 2020. When deep reinforcement learning meets federated learning: Intelligent multitimescale resource management for multiaccess edge computing in 5G ultradense network. IEEE Internet of Things Journal 8, 4 (2020), 2238--2251.

[41]

Zhengxin Yu, Jia Hu, Geyong Min, Zi Wang, Wang Miao, and Shancang Li. 2021. Privacy-preserving federated deep learning for cooperative hierarchical caching in fog computing. IEEE Internet of Things Journal (2021).

[42]

Zhengxin Yu, Jia Hu, Geyong Min, Zhiwei Zhao, Wang Miao, and M Shamim Hossain. 2020. Mobility-aware proactive edge caching for connected vehicles using federated learning. IEEE Transactions on Intelligent Transportation Systems 22, 8 (2020), 5341--5351.

Digital Library

[43]

Chengliang Zhang, Suyi Li, Junzhe Xia, Wei Wang, Feng Yan, and Yang Liu. 2020. {BatchCrypt}: Efficient homomorphic encryption for {Cross-Silo} federated learning. In 2020 USENIX annual technical conference (USENIX ATC 20). 493--506.

[44]

Yupeng Zhang, Lingjie Duan, and Ngai-Man Cheung. 2022. Accelerating Federated learning on non-IID data against stragglers. In 2022 IEEE International Conference on Sensing, Communication, and Networking (SECON Workshops).

[45]

Xin-Ying Zheng, Ming-Chun Lee, and Y-W Peter Hong. 2021. Knowledge Caching for Federated Learning. In 2021 IEEE Global Communications Conference (GLOBECOM). IEEE, 1--6.

Cited By

Cui LWang PHan Y(2024)Edge Caching with Federated Unlearning in Cluster-Centric Small Cell Networks2024 Sixth International Conference on Next Generation Data-driven Networks (NGDN)10.1109/NGDN61651.2024.10744106(37-40)Online publication date: 26-Apr-2024
https://doi.org/10.1109/NGDN61651.2024.10744106

Index Terms

Cache-Enabled Federated Learning Systems
1. Computing methodologies
2. Networks
  1. Network services

Recommendations

Criticality aware tiered cache hierarchy: a fundamental relook at multi-level cache hierarchies
ISCA '18: Proceedings of the 45th Annual International Symposium on Computer Architecture

On-die caches are a popular method to help hide the main memory latency. However, it is difficult to build large caches without substantially increasing their access latency, which in turn hurts performance. To overcome this difficulty, on-die caches ...
Modeling LRU cache with invalidation

Least Recently Used (LRU) is a very popular caching replacement policy. It is very easy to implement and offers good performance, especially when data requests are temporally correlated, as in the case of web traffic.When the data content can change ...
Proxy Cache Algorithms: Design, Implementation, and Performance

Caching at proxy servers is one of the ways to reduce the response time perceived by World Wide Web users. Cache replacement algorithms play a central rolfe in the response time reduction by selecting a subset of documents for caching, so that a given ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MobiHoc '23: Proceedings of the Twenty-fourth International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing

October 2023

621 pages

ISBN:9781450399265

DOI:10.1145/3565287

General Chairs:
Jie Wu,
Suresh Subramaniam,
Program Chairs:
Bo Ji,
Carla Fabiana Chiasserini

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMOBILE: ACM Special Interest Group on Mobility of Systems, Users, Data and Computing

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

NSF (National Science Foundation)
Army Research Office
Singapore University of Technology and Design, and Singapore Ministry of Education

Conference

MobiHoc '23

Sponsor:

SIGMOBILE

MobiHoc '23: Twenty-fourth International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing

October 23 - 26, 2023

DC, Washington, USA

Acceptance Rates

Overall Acceptance Rate 296 of 1,843 submissions, 16%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
766
Total Downloads

Downloads (Last 12 months)568
Downloads (Last 6 weeks)39

Reflects downloads up to 26 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Cui LWang PHan Y(2024)Edge Caching with Federated Unlearning in Cluster-Centric Small Cell Networks2024 Sixth International Conference on Next Generation Data-driven Networks (NGDN)10.1109/NGDN61651.2024.10744106(37-40)Online publication date: 26-Apr-2024
https://doi.org/10.1109/NGDN61651.2024.10744106

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten