Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates
2019-11-20Code Available0· sign in to hype
Cong Xie, Oluwasanmi Koyejo, Indranil Gupta, Haibin Lin
Code Available — Be the first to reproduce this paper.
ReproduceCode
- github.com/xcgoner/AISTATS2020-AdaAlter-GluonNLPOfficialmxnet★ 0
Abstract
When scaling distributed training, the communication overhead is often the bottleneck. In this paper, we propose a novel SGD variant with reduced communication and adaptive learning rates. We prove the convergence of the proposed algorithm for smooth but non-convex problems. Empirical results show that the proposed algorithm significantly reduces the communication overhead, which, in turn, reduces the training time by up to 30% for the 1B word dataset.