Understanding top-k sparsification in distributed deep learning
Distributed stochastic gradient descent (SGD) algorithms are widely deployed in training
large-scale deep learning models, while the communication overhead among workers
becomes the new system bottleneck. Recently proposed gradient sparsification techniques,
especially Top-$ k $ sparsification with error compensation (TopK-SGD), can significantly
reduce the communication traffic without an obvious impact on the model accuracy. Some
theoretical studies have been carried out to analyze the convergence property of TopK-SGD …
large-scale deep learning models, while the communication overhead among workers
becomes the new system bottleneck. Recently proposed gradient sparsification techniques,
especially Top-$ k $ sparsification with error compensation (TopK-SGD), can significantly
reduce the communication traffic without an obvious impact on the model accuracy. Some
theoretical studies have been carried out to analyze the convergence property of TopK-SGD …
Distributed stochastic gradient descent (SGD) algorithms are widely deployed in training large-scale deep learning models, while the communication overhead among workers becomes the new system bottleneck. Recently proposed gradient sparsification techniques, especially Top- sparsification with error compensation (TopK-SGD), can significantly reduce the communication traffic without an obvious impact on the model accuracy. Some theoretical studies have been carried out to analyze the convergence property of TopK-SGD. However, existing studies do not dive into the details of Top- operator in gradient sparsification and use relaxed bounds (e.g., exact bound of Random-) for analysis; hence the derived results cannot well describe the real convergence performance of TopK-SGD. To this end, we first study the gradient distributions of TopK-SGD during the training process through extensive experiments. We then theoretically derive a tighter bound for the Top- operator. Finally, we exploit the property of gradient distribution to propose an approximate top- selection algorithm, which is computing-efficient for GPUs, to improve the scaling efficiency of TopK-SGD by significantly reducing the computing overhead. Codes are available at: \url{https://github.com/hclhkbu/GaussianK-SGD}.
arxiv.org