Training Quantized Nets: A Deeper Understanding
Currently, deep neural networks are deployed on low-power portable devices by first training a full-precision model using powerful hardware, and then deriving a corresponding lowprecision model for efficient inference on such systems. However, training models directly with coarsely quantized weights is a key step towards learning on embedded platforms that have limited computing resources, memory capacity, and power consumption. Numerous recent publications have studied methods for training quantized networks, but these studies have mostly been empirical. In this work, we investigate training methods for quantized neural networks from a theoretical viewpoint. We first explore accuracy guarantees for training methods under convexity assumptions. We then look at the behavior of these algorithms for non-convex problems, and show that training algorithms that exploit high-precision representations have an important greedy search phase that purely quantized training methods lack, which explains the difficulty of training using low-precision arithmetic.
Summary
---------
This paper analyzes the performance of quantized networks at a theoretical level. It proofs convergence results for two types of quantization (stochastic rounding and binary connect) in a convex setting and provides theoretical insights into the superiority of binary connect over stochastic rounding in non-convex applications.
General comments
—-----------------
The paper states that the motivation for quantization is training of neural networks on embedded devices with limited power, memory and/or support of floating-point arithmetic. I do not agree with this argumentation, because we can easily train a neural network on a powerful GPU grid and still do the inexpensive forward-pass on a mobile device. Furthermore, the vast majority of devices does support floating-point arithmetic. And finally, binary connect, which the paper finds to be superior to stochastic rounding, still needs floating-point arithmetic.
Presentation
-------------
The paper is very well written and structured. The discussion of related work is comprehensive. The notation is clean and consistent. All proofs can be found in the (extensive) supplemental material, which is nice because the reader can appreciate the main results without getting lost in too much detail.
Technical section
------------------
The paper’s theoretical contributions can be divided into convex and non-convex results: For convex problems, the paper provides ergodic convergence results for quantization through stochastic rounding (SR) and binary connect (BC). In particular, it shows that both methods behave fundamentally different for strongly convex problems. While I appreciate the importance of those results at a theoretical level, they are limited in their applicability because today’s optimization problems, and neural networks in particular, are rarely convex. For non-convex problems, the paper proofs that SR does not enter an exploitation phase for small learning rates (Theorem 5) but just gets slow (Theorem 6), because its behavior is essentially independent of the learning rate below a certain threshold. The paper claims that this is in contrast to BC, although no formal proof supporting this claim is provided for the non-convex case.
Experiments
-------------
The experimental evaluation compares the effect of different types of quantization on the performance of a computer vision task. The main purpose of the experiments is to understand the impact of the theoretical results in practice.
- First of all, the paper should clearly state the task (image classification) and quantization step (binary). Experimental results for more challenging tasks and finer quantization steps would have been interesting.
- Standard SGD with learning rates following a Robbins-Monro sequence should have been used instead of Adam. After all, the theoretical results were not derived for Adam’s adaptive learning rate, which I am concerned might conceal some of the effects of standard SGD.
- Table 1 feels a little incoherent, because different network architectures were used for different datasets.
- The test error on ImageNet is very high. Is this top-1 accuracy?
- Why only quantize convolutional weights (l.222)? The vast majority of weights are in the fully-connected layers, so it is not surprising that the quantized performance is close to the unquantized networks if most weights are the same.
- A second experiment computes the percentage of changed weights to give insights into the exploration-exploitation behaviour of SR and BC. I am not sure this is a meaningful experiment. Due to the redundancy in neural networks, changing a large numbers of weights can still result in a very similar function. Also, it seems that the curves in Fig. 3(c) do saturate, meaning that only very few weights are changed between epochs 80 and 100 and indicating exploitation even with SR.
Minor comments
-----------------
l.76: Citation missing.
Theorem 1: w^* not defined (probably global minimizer).
l.137: F_i not defined.
l.242: The percentage of changed weights is around 40%, not 50%.
Conclusion
-----------
The paper makes important contributions to the theoretical understanding of quantized nets. Although I am not convinced by the experiments and concerned about the practical relevance, this paper should be accepted due to its strong theoretical contributions.