We proposed Additive Powers-of-Two (APoT) quantization, an efficient nonuniform quantization scheme that attends to the bell-shaped and long-tailed distribution of weights in neural networks. By constraining all quantization levels as a sum of several Powers-of-Two terms, APoT quantization enjoys the overwhelming efficiency of computation and a good match with weights' distribution. A simple reparameterization on the clipping function is applied to generate a betterdefined gradient for updating of optimal clipping threshold. Moreover, weight normalization is presented to refine the input distribution of weights to be more stable and consistent. Experimental results show that our proposed method outperforms state-of-the-art methods, and is even competitive with the full-precision models demonstrating the effectiveness of our proposed APoT quantization. For example, our 5-bit quantized ResNet-50 on ImageNet achieves 76.8% top-1 accuracy without bells and whistles, meanwhile, our model is capable to decrease 31% fixed point computation overhead in uniformly quantized counterpart.
Summary:
The authors propose to compress Neural Networks (NNs) by quantizing their weights to sums of powers-of-two. This allows both to take the non-uniform distribution of the weights into account and to perform fast inference on dedicated hardware. 
Strengths of the paper:
- The problem is clearly stated (in particular the two questions about the clipping operation and the quantization levels are clearly presented in the Introduction). Similarly, the authors alleviate what they call the "rigid resolution" problem by using sums of powers-of-two and clearly explain their choice. 
- The paper addresses the inference time, which is an important metric. Indeed, researchers tend to focus on the size of the compressed weights as this is easily measurable and not questionable. Inference time however depends on the hardware but is crucial as compressed models often that run on embedded devices are often required to run in real-time (say image detection models). The paragraph "Computation" in Section 2.1 is therefore useful for the reader.
Weaknesses of the paper:
- Both the reparameterization of the clipping function and the weight normalization before quantization approaches seem not novel to me, see for instance: "Weight Normalization based Quantization for Deep Neural Network Compression", Cai et al. 
- The experiments are lacking the widely used ResNet-50 baseline (and Table 4 should be in the main paper instead of the appendix). Other tasks such as Image Detection (Mask R-CNN) could also strengthen the impact of the paper. 
- Moreover, unless I am mistaken, the authors "do not quantize the first and last layers". While not quantizing the first layer has no impact on the compressed size (weights of size 7x7x3x64 = 36 KB for a ResNet-18), not quantizing the last layer (ie the classifier, weights of size 512x1000 = 1.95 MB) seems problematic. It is indeed challenging to quantize the classifier, however this adds an overhead of 1.95 MB. Assuming the ResNet18 is compressed with 2 bits/weight, the compressed part has a size of  44.6 MB/16 = 2.8 MB. Thus, with the classifier, its size is 2.8+1.95 = 4.75 MB, which is roughly a x10 compression factor.  
Justification of rating:
The paper presents an interesting idea that accounts both for the inference time and the bell-shaped distribution of weights in NNs. However, the results must be pushed further (see "weaknesses"). 
=== After the rebuttal ===
I thank the authors for their rebuttal. They answer to all the raised concerns with clarity and concision and performed additional experiments. 
In particular, the authors:
- perform additional ResNet50 experiments
- quantize the input layer and the classifier to 8 bits
- clarifying the positioning and in particular with respect to weight normalization
The paper is substantially better in its updated version. Therefore, I am happy to update my rating to "weak accept".