With the rapid growth of data, distributed stochastic gradient descent (DSGD) has been widely used for solving large-scale machine learning problems. Due to the latency and limited bandwidth of network, communication has become the bottleneck of DSGD when we need to train large scale models, like deep neural networks. Communication compression with sparsified gradient, abbreviated as sparse communication, has been widely used for reducing communication cost in DSGD. Recently, there has appeared one method, called deep gradient compression (DGC), to combine memory gradient and momentum SGD for sparse communication. DGC has achieved promising performance in practice. However, the theory about the convergence of DGC is lack. In this paper, we propose a novel method, called global momentum compression (GMC), for sparse communication in DSGD. GMC also combines memory gradient and momentum SGD. But different from DGC which adopts local momentum, GMC adopts global momentum. We theoretically prove the convergence rate of GMC for both convex and non-convex problems. To the best of our knowledge, this is the first work that proves the convergence of distributed momentum SGD (DMSGD) with sparse communication and memory gradient. Empirical results show that, compared with the DMSGD counterpart without sparse communication, GMC can reduce the communication cost by approximately 100 fold with no loss of generalization accuracy. GMC can also achieve comparable (sometimes better) performance compared with DGC, with an extra theoretical guarantee.
Gradient sparsification is an important technique to reduce the communication overhead in distributed training. In this paper, the authors proposed a training method called global momentum compression (GMC) for distributed momentum SGD with sparse gradient. Following existing gradient sparsification techniques such DGC, GMC is also built up on the memory gradient approach; the major distinction between GMC and existing techniques is that GMC keeps track of global gradient to maintain the memory gradient, while the existing technique keeps track of worker-local gradients for memory gradient. The primary contributions in the paper are as the following:
1. The authors propose GMC, a training method for distributed momentum SGD with sparse gradient communication. It uses global gradient (but still achieve sparse communication) to maintain the gradient memory while existing approaches such as DGC use worker-local gradient to do so.
2. The authors prove the convergence rate of GMC for 1. strongly convex and smooth functions 2. convex functions and 3. Non-convex Lipschitz smooth functions. This is the first work on proving the convergence rate of distributed momentum SGD using sparse communication techniques based on memory gradient.
3. Empirically, the authors show that GMC can empirically attain the same model accuracy as conventional distributed momentum SGD with ~100x reduction in communication overhead. It can also match the performance of DGC at the same communication compression rate.
I think in general the ideas and efforts of the authors in proving the convergence rate of distributed momentum SGD with *gradient sparsification* is interesting and important. However, I have the some questions and concerns on validating the claims in the paper. I currently give weak reject but I am happy to raise the score if the authors can clarify or improve in their rebuttal / future drafts. The primary questions and concerns (critical to the rating) are:
1. One important claimed advantage of GMC over existing method is that it uses global gradient for memory gradient, while existing methods such as DGC uses local-work gradient to do so. But I did not find convincing support of this advantage in the paper: Empirically, in the experiment results, I don't think GMC demonstrate better performance than DGC in a statistical meaningful way; instead they are basically demonstrating matching performance. Theoretically, I am not sure if only the global gradient enables the proof of convergence rate while the worker-local gradient cannot. My preliminary feeling is that by bounding the gradient variance, it should also be possible to prove a rate for DGC using worker-local gradient; this is because the difference between the global gradient and the local gradient might be bounded via the gradient variance.
2. In the experiments, the authors focus on momentum SGD for image classification tasks. To better support the versatility and efficacy of GMC, it would be interesting to include some experiments for other domains (e.g. using the STOA transformer style models for NLP tasks). In these models, momentum-like components are also used in the optimizer (e.g. Adam for fairseq for machine translations), it will be interesting to see if the efficacy of GMC also empirically transfer to these settings.
Minor questions (influencing the rating in a secondary way) 
1. Regarding the assumptions in the paper, I think assumption 2 need some validation / support to show that it is a proper one. My preliminary feeling is that assumption 2 is intuitive as the sparsification procedures only zero out small values so that the error introduced in the gradient is small and bounded. But it should be more convincing to empirically show the magnitude of u comparing to the magnitude of gradient g in Equ. 8. 
2. I notice that the experiments uses conventional momentum SGD for a few epochs as warm up, is there any specific reasoning on using this warmup approach instead of the sparsity level warmup as used in DGC?
3. In the experiments, GMC does not use the factor masking trick while DGC uses. If it is for demonstrating the benefits of global gradient for gradient memory, I think it is more proper to also include the results of DGC without factor masking? In this way, this question can be directly answered in an ablation study way by eliminating the possible contribution of using/not using factor masking. 
NITS to improve the paper (not related to the rating):
1. The last contribution bullet forgets to mention that it is about comparing to DGC.
2. In algorithm 1, it is clearer to mention how the mask m is generated (e.g. based on magnitude).
3. In the second paragraph in section 3.2, the vector inner product is not properly written between coefficients and w.
4. Above theorem 1, in the text, the condition for the discussion on the two cases are confusing.
5. In the definition of CR in the first paragraph of section 5, why the summation starts from 5. The text describes as warm up with 5 *epochs* while in the equation it is saying warm up with 4 *steps*.