Acceleration for Compressed Gradient Descent in Distributed Optimization

Abstract

Due to the high communication cost in distributed and federated learning problems, methods relying on sparsification or quantization of communicated messages are becoming increasingly popular. While in other contexts the best performing gradient-type methods invariably rely on some form of acceleration to reduce the number of iterations, there are no methods which combine the benefits of both gradient compression and acceleration. In this paper, we remedy this situation and propose the first accelerated compressed gradient descent (ACGD) methods. In the single machine regime, we prove that ACGD enjoys the rate O((1+)L 1) for -strongly convex problems and O((1+)L) for convex problems, respectively, where L is the smoothness constant and is the variance parameter of an unbiased compression operator. Our results improve upon the existing non-accelerated rates O((1+)L 1) and O((1+)L), respectively, and recover the best known rates of accelerated gradient descent as a special case when no compression (=0) is applied. We further propose a distributed variant of ACGD and establish the rate O(+L +(n+n) L), where n is the number of machines and O hides the logarithmic factor 1 . This improves upon the previous best result O( + L+ Ln ) achieved by the DIANA method. Finally, we conduct several experiments on real-world datasets which corroborate our theoretical results and confirm the practical superiority of our methods.

Tasks

Reproductions