This paper proposes a versatile and powerful training algorithm named Featurelevel Ensemble Effect for knowledge Distillation (FEED), which is inspired by the work of factor transfer. The factor transfer is one of the knowledge transfer methods that improves the performance of a student network with a strong teacher network. It transfers the knowledge of a teacher in the feature map level using high-capacity teacher network, and our training algorithm FEED is an extension of it. FEED aims to transfer ensemble knowledge, using either multiple teacher in parallel or multiple training sequences. Adapting peer-teaching framework, we introduce a couple of training algorithms that transfer ensemble knowledge to the student at the feature map level, both of which help the student network find more generalized solutions in the parameter space. Experimental results on CIFAR-100 and ImageNet show that our method, FEED, has clear performance enhancements, without introducing any additional parameters or computations at test time.
In summary, I think this paper contains some reasonable results based on a reasonable, moderately novel, idea, but unfortunately, it is not yet ready for publication. Reading it made me rather confused. 
Good things:
- The main idea is sensible, though distilling into the same architecture (sFEED) is not that novel. I think the pFEED is probably the more novel part.
- The numerical results are quite good.
- It's a fairly simple method. If others reproduced these results, I think it would be useful.
Problems:
- Some parts of the paper are written in a way that makes the reader confused about what this paper is about. For example the first paragraph. Some motivations I just did not understand.
- Some parts of the paper are repeating itself. For example "introduction" and "related works". The section on related work also includes some quite unrelated papers.
- The references in the paper are often pointing to work that came much later than the original idea or some pretty random recent papers. For example the idea of model compression (or knowledge distillation) is much older than Hinton et al. I believe it was first proposed by Bucila et al. [1] (which the authors mention later as if knowledge distillation and model compression were very different ideas), it definitely doesn't come from Kim et al. (2018). Learning from intermediate representations of the network is at least as old as Romero et al. [2]. Compression into a network of the same architecture is definitely older than Furnarello et al. (2018). It was done, for example, by Geras et al. [3]. The paper also cites Goodfellow et al. (2016) in some pretty random contexts. I don't want to be too petty about references, but unfortunately, this paper is just below a threshold that I would still find acceptable in this respect.
- The comparison in Table 6 would make more sense if the same architectures would be clearly compared. As it is, it is difficult to be certain where the improvement is coming from and how it actually compares to different methods.
Typos: Titap X, ResNext, prarphraser.
References:
[1] Bucila et al. Model Compression. 2006.
[2] Romero et al. FitNets: Hints for Thin Deep Nets. 2014.
[3] Geras et al. Blending LSTMs into CNNs. 2016.