MODEL COMPRESSION VIA DISTILLATION AND QUANTIZATION
Deep neural networks (DNNs) continue to make significant advances, solving tasks from image classification to translation or reinforcement learning. One aspect of the field receiving considerable attention is efficiently executing deep models in resource-constrained environments, such as mobile or embedded devices. This paper focuses on this problem, and proposes two new compression methods, which jointly leverage weight quantization and distillation of larger networks, called "teachers," into compressed "student" networks. The first method we propose is called quantized distillation and leverages distillation during the training process, by incorporating distillation loss, expressed with respect to the teacher network, into the training of a smaller student network whose weights are quantized to a limited set of levels. The second method, differentiable quantization, optimizes the location of quantization points through stochastic gradient descent, to better fit the behavior of the teacher model. We validate both methods through experiments on convolutional and recurrent architectures. We show that quantized shallow students can reach similar accuracy levels to state-of-the-art full-precision teacher models, while providing up to order of magnitude compression, and inference speedup that is almost linear in the depth reduction. In sum, our results enable DNNs for resource-constrained environments to leverage architecture and accuracy advances developed on more powerful devices.
The paper proposes to combine two approaches to compress deep neural networks - distillation and quantization. The authors proposed two methods, one largely relying on the distillation loss idea then followed by a quantization step, and another one that also learns the location of the quantization points. Somewhat surprisingly, nobody has combined the two approaches before, which makes this paper interesting. Experiments show that both methods work well in compressing large deep neural network models for applications where resources are limited, like on mobile devices. 
Overall I am mostly OK with this paper but not impressed by it.  Detailed comments below.
1. Quantizing with respect to the distillation loss seems to do better than with the normal loss - this needs more discussion. 
2. The idea of using the gradient with respect to the quantization points to learn them is interesting but not entirely new (see, e.g., "Matrix Recovery from Quantized and Corrupted Measurements", ICASSP 2014 and "OrdRec: An Ordinal Model for Predicting Personalized Item Rating Distributions", RecSys 2011, although in a different context). I also wonder if it would work better if you can also allow the weights to move a little bit (it seems to me from Algorithm 2 that you only update the quantization points). How about learning them altogether? Also this differentiable quantization method does not really depend on distillation, which is kind of confusing given the title.
3. I am a little bit confused by how the bits are redistributed in the second method, as in the end it seems to use more than the proposed number of bits shown in the table (as recognized in section 4.2). This makes the comparison a little bit unfair (especially for the CIFAR 100 case, where the "2 bits" differentiable quantization is actually using 3.23 bits). This needs more clarification.
4. The writing can be improved. For example, the concepts of "teacher" and "student" is not clear at all in the abstract - consider putting the first sentence of Section 3 in there instead. Also, the first sentence of the paper reads as "... have showed tremendous performance", which is not proper English. At the top of page 3 I found "we will restrict our attention to uniform and non-uniform quantization". What are you not restricting to, then?
Slightly increased my rating after reading the rebuttal and the revision.