Paraphrasing Complex Network: Network Compression via Factor Transfer
Many researchers have sought ways of model compression to reduce the size of a deep neural network (DNN) with minimal performance degradation in order to use DNNs in embedded systems. Among the model compression methods, a method called knowledge transfer is to train a student network with a stronger teacher network. In this paper, we propose a novel knowledge transfer method which uses convolutional operations to paraphrase teacher's knowledge and to translate it for the student. This is done by two convolutional modules, which are called a paraphraser and a translator. The paraphraser is trained in an unsupervised manner to extract the teacher factors which are defined as paraphrased information of the teacher network. The translator located at the student network extracts the student factors and helps to translate the teacher factors by mimicking them. We observed that our student network trained with the proposed factor transfer method outperforms the ones trained with conventional knowledge transfer methods.
This paper aims at transferring information between a complex network and a simpler one by using the so called paraphrase. The idea is having a bridge to transfer knowledge (activations) even when the student (smaller network) has different number of features / feature dimensions compared to the 'teacher'  This paraphaser is trained in an unsupervised manner to map the existing feature maps / information into the desired shape via an autoencoder. Once trained, this information will be included in the loss function when training the student one via a translator (to adapt sizes). This translator is jointly trained with the student and aims at helping the student to "assimilate the information".   - I wonder about the variations with respect k. Seems like this parameter is crucial and fixing this to a value does not consistently provide the best results. What would be a good way to find this value in a different application? For instance table 1 and 3 show a large variation depending on this number.  - the text claims several times on the relevance of the factors extracted. How can this be verified? - Some tables contain a combination of FT+KD and this combination does not always result in better results. What is the take home message here? Why this numbers are not always included? - How does  this compare to FitNets and the idea of helping with intermediate layers? - If the dimensions are the set equal, (line 155) I guess there could be an ablation study showing the need for those translators, right? Would be nice to see those numbers. - In the experimental setup it is mentioned about l2 reg but then the algorithm seems to use l1. The way is written is very confusing.  - In summary, experiments are not really convincing    Other details: - My understanding is table represents a reimplementation of state of the art methods, right? Would be great if the comparison could be directly on published numbers for fair comparison and understanding the strength of the contribution.   - Tables contain redundant information. For instance Table 6 two bottom rows. As the space is limited, this could be omitted.  - Text suggest the proposal also works for object detection. However, numbers do not really support this (marginal increment of 0. 3 - The way results are presented seem to be overclaiming. For instance, Table 2, in my opinion, does not show "much better accuracy" but some improvement.     ------------------------------- I appreciate author's feedback for addressing my concerns. I have increased my score accordingly.