Deep Neural Networks, being memory and computation intensive, are a challenge to deploy in smaller devices. Numerous quantization techniques have been proposed to reduce the inference latency/memory consumption. However, these techniques impose a large overhead on the training procedure or need to change the training process. We present a non-intrusive quantization technique based on re-training the full precision model, followed by directly optimizing the corresponding binary model. The quantization training process takes no longer than the original training process. We also propose a new loss function to regularize the weights, resulting in reduced quantization error. Combining both help us achieve full precision accuracy on CIFAR dataset using binary quantization. We also achieve full precision accuracy on WikiText-2 using 2 bit quantization. Comparable results are also shown for ImageNet. We also present a 1.5 bits hybrid model exceeding the performance of TWN LSTM model for WikiText-2.
The paper is a little hard to follow and some parts are poorly written. While the authors claim that they use the greedy approach (in sec 2) for quantization where both B and \alpha are learned in a greedy way, it is not clear why there is convergence difference between the two as claimed by the authors in section 3.1. Moreover, the authors claimed faster convergence of B than \alpha because fewer bit clips are observed from the left subplot of Figure 2. However, this conclusion is not quite convincing because 1) on the right subplot of Figure 2, it seems that \alpha also becomes more stable after 80k iterations; 2) the fewer bit clips may comes from using a stepwise learning rate decay scheme. Thus, the motivation for using another phase to train the \alpha is not strong.
The iterative quantization approach has limited novelty. It is similar to many quantization methods like BinaryConnect and Xnor-Net, except that the quantization step is not done immediately after the BP and model updates, but after some iterations of full-precision training. Moreover, these methods also use full-precision weights for update during training.
Clarity in the experiments section is a little better than the previous sections. However,
- The proposed method only performs comparably with TTQ, and shows significant accuracy drop on the Cifar-10 and Cifar-100 datasets (especially on Cifar-100)
- On the ImageNet dataset, there is a large accuracy drop of the proposed method compared to Zhou et al. (2018). Though the authors said that they believe their proposed model can reach a higher accuracy by using layer-by-layer quantization as in Zhou et al. (2018), it is hard to verify this claim due to lack of the corresponding results. Thus, efficacy of the proposed method on large datasets or models are hard to evaluate.