Recent work has shown that performing inference with fast, very-low-bitwidth (e.g., 1 to 2 bits) representations of values in models can yield surprisingly accurate results. However, although 2-bit approximated networks have been shown to be quite accurate, 1 bit approximations, which are twice as fast, have restrictively low accuracy. We propose a method to train models whose weights are a mixture of bitwidths, that allows us to more finely tune the accuracy/speed trade-off. We present the "middle-out" criterion for determining the bitwidth for each value, and show how to integrate it into training models with a desired mixture of bitwidths. We evaluate several architectures and binarization techniques on the ImageNet dataset. We show that our heterogeneous bitwidth approximation achieves superlinear scaling of accuracy with bitwidth. Using an average of only 1.4 bits, we are able to outperform state-of-the-art 2-bit architectures.
This paper suggests a method for varying the degree of quantization in a neural network during the forward propagation phase.
Though this is an important direction to investigate, there are several issues:
1. Comparison with previous results is misleading:
a.	1-bit weights and floating point activations: Rastegari et al. got 56.8% accuracy on Alexnet, which is better than this paper 1.4bit result of 55.2%.
b.	Hubara et al. got 51% results on 1-bit weights and 2-bit activations included also quantization first and last layer, in contrast to this paper. Therefore, it is not clear if there is a significant benefit in the proposed method which achieves 51.5% when decreasing the activation precision to 1.4bit. 
Therefore, it is not clear that the proposed methods improve over previous approaches.
2. It is not clear to me: in which dimension of the tensors are we saving the scale factor? If it is per feature map, or neuron, this eliminates the main benefits of quantization: doing efficient binarized operations when doing Weight*activation during the forward pass?
3. The review of the literature is inaccurate. For example, it is not true that Courbariaux et al. (2016) “further improved accuracy on small datasets”: the main novelty there was binarizing the activations (which typically decreased the accuracy). Also, it is not clear if the scale factors introduced by XNOR-Net indeed allowed "a significant improvement over previous work" in ImageNet (e.g., see DoReFA and Hubara et al. who got similar results using binarized weigths and activations on ImageNet without scale factors).  Lastly, the statement “Typical approaches include linearly placing the quantization points” is inaccurate: it was observed that logarithmic quantization works better in various cases. For example, see Miyashita, Lee and Murmann 2016, and Hubara et al.
%%% After Author's Clarification %%%
This paper results seem more positive now, and I have therefore have increased my score, assuming the authors will revise the paper accordingly.