To accelerate the training of machine learning models, distributed stochastic gradient descent (SGD) and its variants have been widely adopted, which apply multiple workers in parallel to speed up training. Among them, Local SGD has gained much attention due to its lower communication cost. Nevertheless, when the data distribution on workers is non-identical, Local SGD requires O(T 3 4 N 3 4 ) communications to maintain its linear iteration speedup property, where T is the total number of iterations and N is the number of workers. In this paper, we propose Variance Reduced Local SGD (VRL-SGD) to further reduce the communication complexity. Benefiting from eliminating the dependency on the gradient variance among workers, we theoretically prove that VRL-SGD achieves a linear iteration speedup with a lower communication complexity O(T 1 2 N 3 2 ) even if workers access non-identical datasets. We conduct experiments on three machine learning tasks, and the experimental results demonstrate that VRL-SGD performs impressively better than Local SGD when the data among workers are quite diverse.
This paper tackles the problem of data-parallel (synchronous) distributed SGD, to optimize the (finite) sum of N non-convex, possibly different (in the so-called non-identical case), loss functions. This paper focuses on improving the communication efficiency compared to several existing methods tackling this problem.
To that end, the authors contribute:
· A novel algorithm and its asymptotic communication complexity.
· The proof that the common metric of the sum over the training steps of the expected squared norm of the gradient at the average of the N parameters is bounded above.
· Experimental results (training loss function of epoch number) comparing this algorithm with 2 existing ones, solving 3 problems under reasonable settings.
  The training time per epoch of VRL-SGD is claimed to be identical to the one of Local SGD, as the algorithm only have minor differences.
- strengths of the paper: 
· The main paper is very easy to follow.
· Good effort to give intuitions on why VLR-SGD can improve the convergence rate of existing algorithms.
  Such an effort is to be highlighted.
· No obvious mistake in the main.   I have not thoroughly checked the full proof though.
-  weaknesses of the paper: 
· The algorithm, while having differences, is quite reminiscent of Elastic Averaging SGD (EASGD) [1].
  Indeed in both algorithms the model update at the workers consists in both descending the local gradient plus descending toward some "moving-average"obtained through averaging all the local models.
  In EASGD, this "moving-average" is common to every worker and the master, which updates it every k steps.
  In this paper, each worker has its own "moving-average", which update computations are different than in EASGD as the use the instant average of the workers' models instead of the previous "moving-average".
[1]Sixin Zhang, Anna Choromanska, Yann LeCun, Deep learning with Elastic Averaging SGD,  NeurIPS, 2015
- Questions I would like the authors to respond to during the rebuttal:
· Could Elastic Averaging SGD (in particular their fastest variant EAMSGD) be applied as-is to solve the non-identical, non-convex optimization problem at hand?
  Despite the authors of EASGD not studying their algorithm in the non-identical case, following what is done in the intuition part of VRL-SGD (in particular Equation (8)), it seems that the update rule of the "moving-average" in EASGD is then equivalent to having a momentumSGD with dampening (instead of the "generalized SGD form" obtained with the approach of VRL-SGD).  Hence my question.
I suggest acceptance. However I'm willing to change my opinion after reading other more qualified reviewers in the sub-area of variance-reduction techniques.
note: If EASGD was to be sound in the non-identical case as well, my decision would not change much.