LOCAL SGD CONVERGES FAST AND COMMUNICATES LITTLE
Mini-batch stochastic gradient descent (SGD) is state of the art in large scale distributed training. The scheme can reach a linear speedup with respect to the number of workers, but this is rarely seen in practice as the scheme often suffers from large network delays and bandwidth limits. To overcome this communication bottleneck recent works propose to reduce the communication frequency. An algorithm of this type is local SGD that runs SGD independently in parallel on different workers and averages the sequences only once in a while. This scheme shows promising results in practice, but eluded thorough theoretical analysis. We prove concise convergence rates for local SGD on convex problems and show that it converges at the same rate as mini-batch SGD in terms of number of evaluated gradients, that is, the scheme achieves linear speedup in the number of workers and mini-batch size. The number of communication rounds can be reduced up to a factor of T 1/2 -where T denotes the number of total steps-compared to mini-batch SGD. This also holds for asynchronous implementations. Local SGD can also be used for large scale training of deep learning models. The results shown here aim serving as a guideline to further explore the theoretical and practical aspects of local SGD in these applications.
The authors of this paper analyze a well known technique for parallel training, where each compute node locally trains a model with SGD, and once in a while the K compute nodes average their models. Local SGD, although not as widely used as mini-batch SGD, can provide some gains in terms of the cost of communication. This can be achieved by decreasing the frequency of synchronization, while locally also increasing the minibatch. 
To the best of my knowledge, the authors are the first to provide a complete theoretical analysis of local SGD for strongly convex functions. They prove that under strong convexity, and the bounded gradients assumption, local SGD will (in the worst case) achieve a linear speedup over vanilla SGD, as long as the parallel models are averaged frequently enough. They show that although frequent averaging is important for speedup, the overall communication cost can be lower than minibatch SGD that may require smaller batches and hence more frequent communication. 
The authors extend their results to the asynchronous case, where a similar convergence bound is derived. The overall theory seems to be partly inspired by the perturbed iterates framework of Mania et al., however the application is novel and interesting.
The authors include some limited experimental results that validate their bounds.
This is a well-written paper, that will certainly be of interest to researchers working on stochastic optimization, and distributed learning. The results are interesting and clearly stated. The proofs seem complete and correct, and are easy to follow. 
I have two minor comments:
1) In a recent paper, Dong et al. [1] suggest that for any problem (convex or nonconvex), the largest possible batch size in minibatch SGD that allows for linear speedups will be proportional to “gradient diversity”, i.e., a measure of similarity between the concurrently processed gradients. For example, when all gradient are identical, there is no speedup to be extracted. This diversity term does not seem to appear in the main theorem, as one may expect. For example, the presented bounds still seem to provide speedup gains for the case where all individual n functions are identical (eg minimum grad. diversity). This should not be possible, as there are no parallel speedups to be extracted in this case. I’m wondering how that fact is reflected in the presented bounds (maybe it’s one of the extreme parameter cases that are not covered by the main theorem).
2) The authors do not provide details of their experimental setup. For example it would be useful to know what hardware they implemented their algorithms on. It seems that they run experiments for up to 1K workers. Are these individual cores, or was this the result of hyper-threading? Finally, it’s unclear if Fig 1 is a theoretical, or an experimental curve.
[1] http://proceedings.mlr.press/v84/yin18a/yin18a.pdf