Distributed stochastic gradient descent (SGD) algorithms are widely deployed in training large-scale deep learning models, while the communication overhead among workers becomes the new system bottleneck. Recently proposed gradient sparsification techniques, especially Top-k sparsification with error compensation (TopK-SGD), can significantly reduce the communication traffic without obvious impact on the model accuracy. Some theoretical studies have been carried out to analyze the convergence property of TopK-SGD. However, existing studies do not dive into the details of Top-k operator in gradient sparsification and use relaxed bounds (e.g., exact bound of Random-k) for analysis; hence the derived results cannot well describe the real convergence performance of TopK-SGD. To this end, we first study the gradient distributions of TopK-SGD during training process through extensive experiments. We then theoretically derive a tighter bound for the Top-k operator. Finally, we exploit the property of gradient distribution to propose an approximate top-k selection algorithm, which is computing-efficient for GPUs, to improve the scaling efficiency of TopK-SGD by significantly reducing the computing overhead.
This paper makes two contributions to gradient sparsification to reduce the communication bottleneck in distributed SGD.
1) Based on an assumption on the distribution of gradient coordinate values that are backed by an empirical study, the paper derives a tighter bound on the approximation quality of top-k gradient sparsification. This result induces better convergence bounds.
2) The authors note that the top-k cannot benefit from the highly parallel architectures popular in ML, and propose an approximate top-k sparsification operator. This operator tries up to three thresholds and checks how many entries are larger than this value. The initial guess is based on approximating the distribution of gradient coordinates with a normal distribution again.
My score is weak reject. I believe that both observations are valid, and that their solutions might be practically meaningful. However, I would like to see the comparison to another baselines [1]. I would also urge the authors to make it more clear that their theoretical results are based on a strong assumption on the distribution of gradients. The top-k approximation algorithm is practical but I find 3-step threshold search inelegant.
Comments: 
1) [1] is another baseline -- compare your method with it too. 
2) 4.3 Convergence performance "operator can select close elements with Top_k" --- It seems obvious that it can select similar elements. The question is whether the number of elements chosen is accurate. I would like to see this evaluated. It is unclear if this scheme is biased. As far as I can see, it might be over- or under-sparsifying. 
3) 3.1 Gradient Distribution "One can easily prove" --- please do so (in the appendix)
4) Theorem 1 - Looks like this can't be true in general. I think it assumes d -> infinity. 
5) 3.1 Gradient Distribution "then pi is a decreasing function" --- should this be pi^2. Also in figure 3, the result of Eqn 7 is only correct if the curve if pi^2.
6) Figure 2: I am not convinced that these distributions are 'gaussian'. In fact, they seem peakier. It seems to me that this should improve the results (i.e. make the descending pi^2 curve more convex). If this is true, I would encourage the authors to discuss this. BTW, the distribution is in terms of the whole model, or just one randomly picked layer?
7) Conclusion "theoretically tighter bound" --- because the assumption on the distribution empirical, I find it slightly misleading to call this a 'theoretical bound'. I would urge the authors to make this very clear. (This does not mean I find the bound meaningless)
8) Introduction: "O(d), which generally limits the system scalability" --- The O(d) does not explain scalability in terms of number of workers as is suggested. Note that even though bandwidth scales with O(d) in all reduce, the latency does scale with n. This should not be ignored.
9) Related work/Gradient Sparsification --- Please add a reference for empirical success of top-k. I am not aware of much use outside of academia.
10) Quite a few language errors (some paragraphs/sentences don't make sense at all and there are many cases with missing 'a's etc.)
11) Some of the experimental details are missing
    - what are the learning rates/batch sizes used in experiments.
    - How topK is performed? Layer-wise or for the full gradient.
    - Table 1 "experimental settings": how these values were chosen.
    - Table 2 --- please define how scaling efficiency is computed.
    - Table 2 --- Do these algorithms achieve the same validation accuracy during the training?
    - Figure 1: which k was used in these plots? 
12) Figure 6: in VGG-16, the gap of 1 percentage point is quite large. This seems expected as the compression ratio is very high (1000x). 
13) Figure 6: Imagenet training scheme is not standard. SOTA validation accuracy for the Imagenet benchmark with Resnet50 is around 76%. Would it also have similar quality loss on later stages as in training VGG or ResNet20 on cifar? 
14) Eqn. 8 - I couldn't follow the first inequality.
15) Introdcution: The first few times "distribution of gradients" is mentioned, it was unclear to me if this was over 'coordinates' (as it seems to be), or over 'data points', or 'training time'. Please clarify.
[1] Jiarui Fang, Cho-Jui Hsieh. Accelerating Distributed Deep Learning Training with Gradient Compression.