Neural network models are resource hungry. Low bit quantization such as binary and ternary quantization is a common approach to alleviate this resource requirements. Ternary quantization provides a more flexible model and often beats binary quantization in terms of accuracy, but doubles memory and increases computation cost. Mixed quantization depth models, on another hand, allows a trade-off between accuracy and memory footprint. In such models, quantization depth is often chosen manually (which is a tiring task), or is tuned using a separate optimization routine (which requires training a quantized network multiple times). Here, we propose Smart Ternary Quantization (STQ) in which we modify the quantization depth directly through an adaptive regularization function, so that we train a model only once. This method jumps between binary and ternary quantization while training. We show its application on image classification.
The paper discusses a generalization to low bit quantization and combines the approaches of binary and ternary quantization methods. Past methods such as Binary Connect and Binary Weights Network have shown that you can train a network efficiently with 1-bit quantization, and methods such as Ternary Weights Network demonstrate 2-bit quantization with weights taking one of {-1, 0, 1} * mu, with mu being a scale computed per weight tensor. The authors generalize these two methods so that the choice of binary vs ternary weights can be made per layer automatically during training. The primary contribution to make that work is by incorporating a generic regularizer with addition hyper-parameters to trade-off between the binary weight regularizer and ternary weight regularizer. In addition to that, the regularization also includes a prior to make the layers prefer binary weights by default. This is done by adding a cost that penalizes the choice of ternary weights for each layer.
Overall, the paper is well written and explained, with supporting experiments to show on MNIST and on CIFAR10 that this method performs quite competitively compared to an all-binary or all-ternary weights network. Low-bit quantization is an important research area and this paper makes a strong contribution by studying mixed-precision low-bit quantization. The mathematical explanations of the generalized regularizer are well justified. For instance, the regularization constant lambda is normalized for each layer by the total number of weights to evenly weight all layers.
Although the experiments cover MNIST and CIFAR10, it's not clear how mixed-precision low-bit methods perform on models more prevalent in the real-world. Supporting the experiments with ResNet variants on ImageNet would help clarify that further.  The paper very well explains the fundamentals of 1-bit and 2-bit methods, and the contribution of this paper (generalization of these two methods) is a somewhat natural extension without significant novelty. Moreover, in addition to the accuracy, it would also be better to understand how the run-time performance of such models (on existing software and hardware implementations) compares to pure binary and ternary networks, as knowledge of that would reveal insights into systems optimizations to be made in future work in this field. Given all of this, my rating is a weak accept.
Pros:
- The problem and the fundamentals (prior work) are very well laid out.
- The regularization component that generalizes the two types of quantization is sound.
- Experiments on CIFAR10 strongly show improved accuracy and higher compression ratio compared to ternary weights network.
Cons:
- Lacking more realistic experiments on larger datasets such as ImageNet and models like ResNets.
- The runtime performance of how STQ compares to 1-bit and 2-bit quantization variants isn't shown.
- The regularizer introduces more hyper-parameters to tweak and it's not clear how sensitive these are to the choice of the architecture. Different values of gamma are used in the two experiments, and further analysis on how the performance varies for various values of lambda and gamma would shed further insight.
Minor comments:
- In equation (3), the term under argmin should be mu and not alpha.
- Section 3, line above equation (9) reads "very per layer" instead of "vary per layer"