Accelerating distributed stochastic gradient descent with adaptive periodic parameter averaging: poster

Training of large-scale machine learning models presents a hefty communication challenge to the Stochastic Gradient Descent (SGD) algorithm. In a distributed computing environment, frequent exchanges of gradient parameters between computational ...$\mathcal{}\mathrm{}$

Stochastic gradient descent is popular for large scale optimization but has slow convergence asymptotically due to the inherent variance. To remedy this problem, we introduce an explicit variance reduction method for stochastic gradient descent which we ...

The method of conjugate gradients provides a very effective way to optimize large, deterministic systems by gradient descent. In its standard form, however, it is not amenable to stochastic approximation of the gradient. Here we explore ideas from ...