Neural network quantization has an inherent problem called accumulated quantization error, which is the key obstacle towards ultra-low precision, e.g., 2-or 3-bit precision. To resolve this problem, we propose precision highway, which forms an end-to-end high-precision information flow while performing the ultralow-precision computation. First, we describe how the precision highway reduce the accumulated quantization error in both convolutional and recurrent neural networks. We also provide the quantitative analysis of the benefit of precision highway and evaluate the overhead on the state-of-the-art hardware accelerator. In the experiments, our proposed method outperforms the best existing quantization methods while offering 3-bit weight/activation quantization with no accuracy loss and 2-bit quantization with a 2.45 % top-1 accuracy loss in ResNet-50. We also report that the proposed method significantly outperforms the existing method in the 2-bit quantization of an LSTM for language modeling.
This paper studies methods to improve the performance of quantized neural networks.  The paper is largely centered around the idea of "precision highways" (full-precision residual connections) that run in parallel to fully-quantized convolutions.  However, the paper also throws in a toolbox of other methods like distillation from a teacher network, a quantization method based on the Laplace distribution, and a fine tuning scheme.
The paper reports performance for the resulting networks that is impressive but still believable.   They also do very extensive experiments, including an ablation study in Table 1 that I really liked, and a study of how the precision of the skip connections impacts overall performance.   I also like the visualizations of how quantization impacts the loss surface.
My main concern about this paper is that is has conceptual overlap with other approaches.  The authors are not the first to quantize resnets, and other papers have looked at teacher training and distillation as a method of refinement.  The authors are fairly upfront about this though, and I think this paper is the first to do a really thorough investigation of the impacts of skip connections in their own right.    Realistically, fully binarizing neural nets without modification is unlikely to lead to good performance.  The idea of leaving the skip connections with higher precision is a good compromise that achieves hardware friendliness along with strong performance, so I think it's worth having a paper like this that takes a closer look at this approach.
A few questions I had:
1)  I can't tell exactly what methods are being used in Table 1.  When the "highway" box is unchecked, does this mean the skip connection is absent?  Or that it exists but with full precision?  Or maybe that the skip connection branched after the quantization instead of before?   Also, what fine-tuning methods is used when the "teacher" box is un-checked?
2) You implemented your own version of Zhuang's method.  However, I'd like to know how your numbers compare to the original reported numbers in Zhuang's paper.
One other minor criticism - When you fine-tune a modified network, the activations and weights will change.  It could be that the networks is modifying its parameters to account for (i.e., cancel out) the quantization errors.  For this reason I don't interpret Figure 4 as evidence for accumulation of error.  Perhaps this type of behavior would exists if you fine-tuned two full-precision networks using different random seeds, or different teacher networks.