Accompanying with the flourish development in various fields, deep neural networks, however, are still facing with the plight of high computational costs and storage. One way to compress these heavy models is knowledge transfer (KT), in which a light student network is trained through absorbing the knowledge from a powerful teacher network. In this paper, we propose a novel knowledge transfer method which employs a Student-Teacher Collaboration (STC) network during the knowledge transfer process. This is done by connecting the front part of the student network to the back part of the teacher network as the STC network. The back part of the teacher network takes the intermediate representation from the front part of the student network as input to make the prediction. The difference between the prediction from the collaboration network and the output tensor from the teacher network is taken into account of the loss during the train process. Through back propagation, the teacher network provides guidance to the student network in a gradient signal manner. In this way, our method takes advantage of the knowledge from the entire teacher network, who instructs the student network in learning process. Through plentiful experiments, it is proved that our STC method outperforms other KT methods with conventional strategy.
The paper suggests a new method for knowledge transfer from teacher neural network to student: student-teacher collaboration (STC). The idea is that teacher not only provides student with the correct answer or correct intermediate representation, but also instructs student network on how to get the right answer. The paper suggests to merge the front part of the student network and back part of the teacher network into the collaboration network. In this way, the weights in the student subnetwork are learned from the loss backpropagated from the teacher subnetwork. The target labels for collaboration network are outputs of teacher network. The method is adapted for different tasks, classification and bounding box regression are presented in the experiments. It outperforms previous methods on various datasets. Furthermore, the method shows good results when integrated with traditional knowledge distillation.
Overall, the paper is a significant algorithmic contribution. The related work section provides thorough review of different methods to decrease computational costs, including not only knowledge distillation, but also pruning, compressing and decomposition approaches. The idea is elegant and, to the best of my knowledge, has never been suggested in other works. Considering the theoretical part, it is clearly shown how the gradient signal from the teacher sub-network guides the student network on which part of the weights should be paid attention on. All derivations are easy to follow. The paper also considers how the suggested idea is aimed to solve the problems of previous knowledge transfer methods. The experimental section is consistent and clearly shows the advantage of the suggested method. Teacher and student networks used are different sizes of ResNet, Wide ResNet and VGG. The paper presents classification experiments on CIFAR-10, CIFAR-100, ImageNet and object detection experiment on PASCAL VOC 2007 dataset. STC outperforms previous methods, both with KD integration and without. The performance is always better than pure student training (which was not always the case for previous methods) and sometimes the results are even better than teacher performance. Finally, the choice of teacher output as target over soft target and ground truth, which was previously motivated in the theoretical section, is shown to be superior in the experiment.
Possible improvement of the paper is the instruction on how to choose the intermediate layer from where to teach the representation, i.e. where the student sub-network ends and teacher sub-network begins. For object detection experiment the choice of the border is naturally defined by the architecture of the network in Faster-RCNN approach. Could the choice be different? May be somewhere inside the BackBone part of the networks? For classification, it could be interesting to study how this choice influences the results. However, this question didn’t affect my score of the paper, and, as far as I know, it is also not considered in the previous works on knowledge distillation.
Minor comments
1.	In the context of accelerating the models using decomposition, Lebedev et al., ICLR 2015 could be cited.
2.	Page 2: difference tasks -> different tasks
3.	Page 2 the first bullet point: additionally utilizes -> additionally utilizing/which additionally utilizes
4.	Page 2 the third bullet point: brings good generalizability -> which brings good generalizability
5.	Page 5: “training strategy is more accelerate than” – odd phrase
6.	Page 6: while KT has conflicts with KD in some cases -> while FT has conflicts with KD in some cases.