A Linear Speedup Analysis of Distributed Deep Learning with Sparse and Quantized Communication
The large communication overhead has imposed a bottleneck on the performance of distributed Stochastic Gradient Descent (SGD) for training deep neural networks. Previous works have demonstrated the potential of using gradient sparsification and quantization to reduce the communication cost. However, there is still a lack of understanding about how sparse and quantized communication affects the convergence rate of the training algorithm. In this paper, we study the convergence rate of distributed SGD for non-convex optimization with two communication reducing strategies: sparse parameter averaging and gradient quantization. We show that O(1/ p MK) convergence rate can be achieved if the sparsification and quantization hyperparameters are configured properly. We also propose a strategy called periodic quantized averaging (PQASGD) that further reduces the communication cost while preserving the O(1/ p MK) convergence rate. Our evaluation validates our theoretical results and shows that our PQASGD can converge as fast as full-communication SGD with only 3% 5% communication data size.
======== after rebuttal Some of the concerns I had about the paper have been addressed by the authors' response (eg in relation to convergence proofs), yet some still stand. Especially the ones related to experiments, and parameter tuning. Since the authors appropriately responded to some of my key criticisms I will upgrade my score from 5 to 6.   ========original review  This paper analyzes the convergence rate of distributed mini-batch SGD using sparse and quantized communication with application to deep learning. Based on the analysis, it proposes combining sparse and quantized communication to further reduce the communication cost that burdens the wall clock runtime in distributed setups.  The paper is generally well written. The convergence analysis does appear to make sense, and the proposed combination of sparsification and quantization seems to save runtime a little bit with proper parameter tuning.  However, I have a few concerns about this work, listed below:  - A concern about this work is the novelty of the analysis. The key idea of proving convergence with noisy gradient update is bounding the variance of the gradient while assuming the noisy gradient update is unbiased, which, has been known for a long time, eg see [1-3] below. In fact, the following two papers, although not discussed in the submitted paper, have given detailed ways of minimizing variance for sparsified+quantized gradients.  [1] Jakub Konečný and Peter Richtárik, Randomized Distributed Mean Estimation: Accuracy vs Communication, 2016. [2] Ananda Theertha Suresh et al, Distributed Mean Estimation with Limited Communication, 2017. [3] Jianqiao Wangni et al. Gradient sparsification for communication-efficient distributed optimization. arXiv preprint arXiv:1710.09854, 2017.  Given the above work, it is unclear what are the new insights provided by this paper's theoretical analysis. One can simply prove bounds on the Variance of the quant./sparse gradients (assuming they are unbiased), and a convergence rate can be shown in a plug an play way. As such, it is important to compare the variance of the proposed techniques, for a given communication budget, in comparison to prior art, as listed above.  - Furthermore, it is not clear why the proposed technique, i.e., using a combination of sparsification and quantization, will be significantly better than standalone sparsification or quantization. As the theoretical convergence bound in the paper (line 255) suggests, the proposed approach converge at rate O(1+q/ sqrt(MK) ), which is worse than that using sparse communication (O(1/sqrt(MK) ), line 173) and that of using quantized communication (O(1/sqrt(MK) ), line 211). The authors do argue that when the parameters are well chosen, the rate can be roughly the same, but this does not explain why a combination can provide an improvement. I might have missed something on that part of the text.   - A key challenge in understanding the performance of distributed SGD using sparse and quantized Communication is how the speedup changes as the compression ratio (either quant. or sparsification) changes. That is for epsilon accuracy, how faster does training happen on P machines versus 1, and what is the best selection of sparsity/quantization levels? Unfortunately, there is no detailed analysis or any experiment shown in the main paper about this. It is critical to have such results (at least an empirical study) to understand the convergence rate in terms of gradient compression efficiency.  - In their experimental section, the authors compare the proposed technique with QSGD, terngrad, and vanilla sgd on a specific set of parameters. Where the parameters all tuned for all methods? Eg were the learning rates optimized individually for both QSGD and terngrad, and all setups of compute nodes?  It is not clear if the speedup will be significant with well-chosen parameters for QSGD and terngrad, and this is necessary for a fair comparison with prior art.  I overall think that the ideas presented in this paper are useful and interesting, but some work still needs to be done in order to clarify some points of confusion. My most major concern is how the proposed algorithm performs experimentally when all other algorithms (eg QSGD) operate with tuned hyperparameters. Finally, since the paper claims to establish linear speedups, these should be also proved experimentally in comparison to prior art.