In distributed training, the communication cost due to the transmission of gradients or the parameters of the deep model is a major bottleneck in scaling up the number of processing nodes. To address this issue, we propose dithered quantization for the transmission of the stochastic gradients and show that training with Dithered Quantized Stochastic Gradients (DQSG) is similar to the training with unquantized SGs perturbed by an independent bounded uniform noise, in contrast to the other quantization methods where the perturbation depends on the gradients and hence, complicating the convergence analysis. We study the convergence of training algorithms using DQSG and the trade off between the number of quantization levels and the training time. Next, we observe that there is a correlation among the SGs computed by workers that can be utilized to further reduce the communication overhead without any performance loss. Hence, we develop a simple yet effective quantization scheme, nested dithered quantized SG (NDQSG), that can reduce the communication significantly without requiring the workers communicating extra information to each other. We prove that although NDQSG requires significantly less bits, it can achieve the same quantization variance bound as DQSG. Our simulation results confirm the effectiveness of training using DQSG and NDQSG in reducing the communication bits or the convergence time compared to the existing methods without sacrificing the accuracy of the trained model.
Authors establish a connection between communication reduction in distributed optimization and dithered quantization. This allows us to understand prior approaches in a new perspective, and also motivates authors to develop two new distributed training algorithms which communication overhead is significantly reduced. The first algorithm, DQSG, uses dithered quantization to reduce the communication bits. The second algorithm, NDQSG, uses nested dithered quantization to further reduce the amount of needed communication. The usefulness of these algorithms are empirically validated by computing the raw communication bits and average entropy of them. Therefore, dithered communication seems to provide both theory and algorithm which are useful.
The paper is clearly written. It provides a succinct review of dithered quantization and previous works, and figures provide a good insight into why the algorithm works, especially Figure 3.
Theorems in this paper are mostly about plugging in properties of dithered quantization into standard results in stochastic optimization, but they are still useful. The analysis of NDQSG does not seem to be as complete as that of DQSG, however. With NQSG, now workers are divided into two groups, and there would be an interesting tradeoff between assignments to these two: how should we balance two groups? This might be tricky to analyze, but it is still useful to clarify limitations and provide conjectures. At least, this could be analyzed empirically.
pros:
* establishing a connection to other topic of research often facilitates productive collaboration between two fields
* provides a new perspective to understand prior work
* provides new useful algorithms
cons:
* experiments were conducted on small models and small datasets
* unclear models are large enough to demonstrate the need for communication reduction; in other words, it is unclear wall-time would actually be reduced with these algorithms.