Computer Science > Machine Learning
[Submitted on 4 Oct 2023 (v1), last revised 24 May 2024 (this version, v3)]
Title:FedAWARE: Maximizing Gradient Diversity for Heterogeneous Federated Server-side Optimization
View PDF HTML (experimental)Abstract:Federated learning (FL) is a distributed learning framework where numerous clients collaborate with a central server to train a model without sharing local data. However, the standard federated optimization in real-world applications faces both statistical and system heterogeneity challenges, which result in unfavorable convergence behavior. The previous works attempted to modify the local training process (client-side) to tackle heterogeneity challenges. However, they ignored that the updates on the server side can coordinate the diverse local updates efficiently. This work explores the effect of server-side updates against heterogeneity issues. We first introduce the gradient diversity maximization direction findings, suggesting the global model moves continuously in this direction for fast and stable convergence. Then, we derive a novel server-side optimizer \textsc{FedAWARE} with rigorous convergence analysis for general non-convex settings. Our extensive experiments across multiple heterogeneous federated settings using four datasets showcase that \textsc{FedAWARE} achieves competitive convergence performance in comparison to state-of-the-art adaptive federated optimizers. Furthermore, our results show that \textsc{FedAWARE} can enhance the performance of FL algorithms as a plug-in module. Our source code is available at \url{this https URL}.
Submission history
From: Dun Zeng [view email][v1] Wed, 4 Oct 2023 10:15:57 UTC (3,491 KB)
[v2] Sun, 4 Feb 2024 18:33:24 UTC (3,867 KB)
[v3] Fri, 24 May 2024 16:13:22 UTC (6,959 KB)
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
IArxiv Recommender
(What is IArxiv?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.