Knowledge distillation (KD) is a popular method for reducing the computational overhead of deep network inference, in which the output of a teacher model is used to train a smaller, faster student model. Hint training (i.e., FitNets) extends KD by regressing a student model's intermediate representation to a teacher model's intermediate representation. In this work, we introduce bLock-wise Intermediate representation Training (LIT), a novel model compression technique that extends the use of intermediate representations in deep network compression, outperforming KD and hint training. LIT has two key ideas: 1) LIT trains a student of the same width (but shallower depth) as the teacher by directly comparing the intermediate representations, and 2) LIT uses the intermediate representation from the previous block in the teacher model as an input to the current student block during training, avoiding unstable intermediate representations in the student network. We show that LIT provides substantial reductions in network depth without loss in accuracy -for example, LIT can compress a ResNeXt-110 to a ResNeXt-20 (5.5×) on CIFAR10 and a VDCNN-29 to a VDCNN-9 (3.2×) on Amazon Reviews without loss in accuracy, outperforming KD and hint training in network size for a given accuracy. We also show that applying LIT to identical student/teacher architectures increases the accuracy of the student model above the teacher model, outperforming the recently-proposed Born Again Networks procedure on ResNet, ResNeXt, and VDCNN. Finally, we show that LIT can effectively compress GAN generators.
This paper introduces LIT, a network compression framework, which uses multiple intermediate representations from a teacher network to guide the training of a student network. Experiments are designed such that student networks are shallower than teacher networks, while maintaining their width. The method is validated on CIFAR-10 and 100 as well as on Amazon Reviews.
The paper is clearly written and easy to follow. The main novelty of the paper is essentially using the teacher intermediate representations as input to the student network to stabilize the training, and applying the strategy to recent networks and tasks.
The authors claim that they are only concerned with knowledge transfer between layers of the same width, that is teacher and student network been designed (by model construction) to have the same number of downsampling operations, while maintaining the same number of stages (referred to as sections in the paper). However, resnet-based architectures have been shown to perform iterative refinement of their features between downsampling operations (see e.g. https://arxiv.org/pdf/1612.07771.pdf and https://arxiv.org/pdf/1710.04773.pdf ). Moreover, these models were also shown to be good regularizers, since they can reduce their model capacity as needed (see https://arxiv.org/pdf/1804.11332.pdf).  Therefore, having experiments skipping stages would be interesting, and may allow to further compress the networks (by skipping layers or stages which do not incorporate much transformation). Following https://arxiv.org/pdf/1804.11332.pdf, for the sake of completeness, it might also be interesting to compare LIT results to the ones obtained by just removing layers in the teacher network which have small weight norms.
In method, the last sentence before "knowledge distillation loss" suggests the training of student networks might not be done end-to-end. Could the authors clarify this?
It seems there might be a typo in the KD loss of "knowledge distillation loss", equation (2). Shouldn't the second term of the equation be a function of p^T and q^T (with temperature)?
I would suggest changing "sections" to stages, as previously introduced in https://arxiv.org/pdf/1612.07771.pdf .
As for the experiments, it would be more interesting to see this kind of analysis on ImageNet (pretained resnet models are readily available).
Figure 3, why not add hint training as well?
Figure 4, what's the dataset used here?
In Section 4.2, it seems that the choice of the IR layer in the analysis could have a significant impact. How was the layer chosen for the ablation study experiments?
There are a few overstatements in the paper:
- page 5, paragraph 2: FitNets proposes a general framework to transfer knowledge from a teacher network to a student network through intermediate layers. Thus, the framework itself does not require the student networks to be deeper and thinner than the teacher network.
- page 6, "LIT can compress GANs": authors claim to overcome limitations of KD when it comes to applying knowledge transfer to pixel-wise architecture that do not output distributions. It seems that changing the loss and using a l2 loss instead is a rather minor change, especially since performing knowledge transfer by means of l2 (although at intermediate layers) has already been explored in FitNets.
Please add references for inception and FID scores.
Please fix references format in page 10.