ANALYSIS OF QUANTIZED MODELS
Deep neural networks are usually huge, which significantly limits the deployment on low-end devices. In recent years, many weight-quantized models have been proposed. They have small storage and fast inference, but training can still be time-consuming. This can be improved with distributed learning. To reduce the high communication cost due to worker-server synchronization, recently gradient quantization has also been proposed to train deep networks with full-precision weights. In this paper, we theoretically study how the combination of both weight and gradient quantization affects convergence. We show that (i) weight-quantized models converge to an error related to the weight quantization resolution and weight dimension; (ii) quantizing gradients slows convergence by a factor related to the gradient quantization resolution and dimension; and (iii) clipping the gradient before quantization renders this factor dimension-free, thus allowing the use of fewer bits for gradient quantization. Empirical experiments confirm the theoretical convergence results, and demonstrate that quantized networks can speed up training and have comparable performance as full-precision networks.
Summary:
This paper studies the convergence properties of loss-aware weight quantization with different gradient precisions in the distributed environment, in which servers keeps the full-precision weights and workers keeps quantized weights. The authors provided convergence analysis for weight quantization with full-precision, quantized and quantized clipped gradients. Specifically, they find that: 1) the regret of loss-aware weight quantization with full-precision gradient converge to an error related to the weight quantization resolution and dimension d. 2) gradient quantization slows the convergence by a factor related to gradient quantization resolution and dimension d. 3) gradient clipping renders the speed degradation dimension-free. 
Comments:
Pros:
- The paper is generally well written and organized. The notation is clean and consistent. Detailed proofs can be found in the appendix, the reader can appreciate the main results without getting lost in details.
- The paper provides theoretical analysis for the convergence properties of loss-aware weight quantization with full-precision gradients, quantized gradient and clipped quantized gradient, which extends existing analysis beyond full-precision gradients, which could be useful for distributed training with limited bandwidth. 
Cons:
- It is unclear what problems the authors try to solve. The problem is about gradient compression, or how the gradient precision will affect the convergence for training quantized nets in the distributed environment, in which workers have limited computation power and the network bandwidth is limited. It is an interesting setting, however, the author does not make it clear the questions they are asking and how the theoretical results can guide the practical algorithm design. 
- The authors mentioned that quantized gradient slows convergence (relative to using full-precision gradient) in contribution 2 while also claims that quantizing gradients can significantly speed up training of quantized weights in contribution 4, which is contradictory to each other.
- It is not clear what relaxation was made on the assumptions of f_t in section 3.1. The analysis are still based on three common assumptions: 1) f_t is convex 2) f_t is twice differentiable 3) f_t has bounded gradients. The assumptions and theoretical results may not hold for non-convex deep nets. E.g., the author does not valides the theorems results on d with neural networks but only with linear models in section 4.1.
- The author demonstrate training quantized nets in the distributed environment with quantized gradients, however, no comparison is made with other related works (e.g., Wen et al, 2017). 
Questions: 
- Theorem 1 is an analysis for training with quantized weights and full-precision gradients, which is essentially the same setting as BinaryConnect. Similar analysis has been done in Li et al, 2017. What is the difference or connection with their bound?
- It is not clear how gradienta are calculated w.r.t. quantized weights on worker, is straight through estimator (STE) used for backpropagation through Q_w?
- In section 3.3, why is \tilde{g}_t stochastically quantized gradient? How about statiscally quantized gradients?
- Why do the authors use linear model in section 4.1? Why are the solid lines in Figure 3 finished earlier than dashed lines? For neural networks, a common observation is that the larger the dimension d, the better the generalization performance. However, Figure 3 and Theorem 1 seem to be contradictory to this common belief. Would it possible to verify the theorem on deep nets of different dimension?
- Why does the number of worker affect the performance? I failed to see why the number of workers affect the performance of training if it is a synchronized distributed training with the same total batch size. After checking appendix C, I think it is better to discuss the influence of batch sizes rather than the number of workers.
- Why is zero weight decay used for CIFAR-10 experiment but non-zero weight decay for imagenet experiment? How was weight decay applied in Adam for quantized weights? 
Minor issues: 
- The notation of full-precision gradient w.r.t quantized weights in Figure 1 should be \hat{g}_t, however, g_t is used.