Jan 29, 2024 · We propose a new approach called CO2 that introduces local-updating and asynchronous communication to the distributed data-parallel training.

CO2: Efficient Distributed Training with Full Communication-Computation Overlap (Paper: https://arxiv.org/abs/2401.16265).

By selecting appropriate local updating step τ in accordance with the communication environment, we can achieve full overlap of model parameter synchronization.

CO2 even performs well on low-bandwidth large clusters, increasing tau to enable more overlap. 3. With staleness gap penalty and outer momentum clipping, CO2 ...

We propose a new approach called CO2 that introduces local-updating and asynchronous communication to the distributed data-parallel training, thereby ...

Jan 29, 2024 · Our proposed CO2 introduces asynchronism to overlap parameter synchronization and local computation, achieving 100% scalability with good ...

Apr 8, 2024 · Our new work, just accepted by ICLR 2024 Spotlight, CO2: Efficient Distributed Training with Full Communication-Computation Overlap.

With a big enough setting of localsgd_frequency , the communication is able to be fully overlapped by local computation steps. CO2DistributedDataParallel takes ...

Mar 20, 2024 · CO2: Efficient Distributed Training with Full Communication-Computation Overlap · Training Neural Networks from Scratch with Parallel Low-Rank ...

CO2: Efficient distributed training with full communication-computation overlap ... Various Lengths, Constant Speed: Efficient Language Modeling with ...