Heterogeneous Bitwidth Binarization in Convolutional Neural Networks
Recent work has shown that fast, compact low-bitwidth neural networks can be surprisingly accurate. These networks use homogeneous binarization: all parameters in each layer or (more commonly) the whole model have the same low bitwidth (e.g., 2 bits). However, modern hardware allows efficient designs where each arithmetic instruction can have a custom bitwidth, motivating heterogeneous binarization, where every parameter in the network may have a different bitwidth. In this paper, we show that it is feasible and useful to select bitwidths at the parameter granularity during training. For instance a heterogeneously quantized version of modern networks such as AlexNet and MobileNet, with the right mix of 1-, 2-and 3-bit parameters that average to just 1.4 bits can equal the accuracy of homogeneous 2-bit versions of these networks. Further, we provide analyses to show that the heterogeneously binarized systems yield FPGA-and ASIC-based implementations that are correspondingly more efficient in both circuit area and energy efficiency than their homogeneous counterparts.
The paper makes an observation that in general 2-bit models are more accurate than 1-bit model. The goal in this paper is to make the performance of lower-bit models closer to than of without (significantly) compromising on their efficiency. The paper achieves this by binarizing to a mixture of bitwidths as opposed to the traditional approach of a uniform bitwidth to all values. In particular, it extends the approach presented in Tang et al. (2017) that homogeneously binarizes to n bits. The paper proposes to examine individual values and decide whether to binarize them (using the Mask Tensor M). The paper proposes an algorithm to achieve an average bidwidth of B (non-integer) by taking as input a targeted distribution over individual bitwidths, P. In order to achieve this target bitwidth distribution, the algorithm greedily assigns low bitwidths to values that are closer to average values and so are likely to be accurate without employing additional bits and vice versa. The experiments are conducted using AlexNet and MobileNet using ImageNet data.  The paper is well-written and easy to follow. The method is somewhat of a simple extension of the idea presented in previous work but the end-results are promising.   I understand that the algorithm not only takes the desired average number of bits (B) as input but also the distribution over individual bitwidths (P). First of all, this is not clear in Algorithm 1, and should be explained. From the algorithm's description it seems that the only input is B (apart from other things).   How sensitive is the algorithm to the choice of P? Ideally, the authors should have shown the variation in performance for different desired distributions (all yielding the same value of average bitwidth B). However, I am willing to listen to authors' opinion on this. Also, are there any recommendations/heuristics to choose a good P?  Minor comment: the sentence in line 26 is difficult to read.  I have read the author response. Thanks for your clarifications. Please include them in your next version.