Knowledge Distillation by On-the-Fly Native Ensemble
Knowledge distillation is effective to train the small and generalisable network models for meeting the low-memory and fast running requirements. Existing offline distillation methods rely on a strong pre-trained teacher, which enables favourable knowledge discovery and transfer but requires a complex two-phase training procedure. Online counterparts address this limitation at the price of lacking a high-capacity teacher. In this work, we present an On-the-fly Native Ensemble (ONE) learning strategy for one-stage online distillation. Specifically, ONE only trains a single multi-branch network while simultaneously establishing a strong teacher on-the-fly to enhance the learning of target network. Extensive evaluations show that ONE improves the generalisation performance of a variety of deep neural networks more significantly than alternative methods on four image classification dataset: CIFAR10, CIFAR100, SVHN, and ImageNet, whilst having the computational efficiency advantages.
Summary:  Authors propose a novel multi-branch network with a loss function that uses distillation from a combined branch to distill into individual branches. The technique is motivated by the idea that Teacher-Student knowledge distillation is a two-step process often requiring a large pre-trained teacher. Their method builds a teacher, out of weighted ensemble and uses that to train the network. They are able to show that the combined network (ONE-E) is far superior to standalone networks, and the individual branch (ONE) is also better than its counterpart (i.e if it were trained without any of the loss functions and the branches).  Pros: 1. Excellent write-up This is a very well written paper. Literature study is comprehensive and the whole paper is easy to follow even if it looks dense in contentwise.  2. Intuitive and excellent idea The idea presented is both intuitive and excellent. People have been using an ensemble of models for various tasks, and it is well known that ensemble will give better results as long as there is some variance in the model.  Another approach to improve the accuracy of the model is to train it with soft-logits of a bigger model. They were able to take advantage of both the techniques (ensemble and knowledge distillation) in a single setup.  3. Excellent ablation study They have presented their technique, as a network modification (adding branches, sharing layers) and change is an objective function (KL div loss + CE). They show the result of each of these techniques individually (Table 5) and show that the combination of all of these provides the best result. This leaves no doubt to the reader on the importance of each of the changes.  4. Superior results Their results on CIFAR/SVHN/Imagenet all look very good and promising. It looks like the technique improves the result for various types of model, thus it is agnostic to the structure (Table 1). It outperforms other ensemble techniques (Table 4) which are more computationally expensive than the presented ONE technique.  Cons:  1. Section 4.5 feels incomplete and an after-thought. I understand that space was a huge limitation, however, if possible, I would like to see a better analysis of why this works compared to other ensemble and Mixture of Expert techniques. Ensembles work when the models have a variance between them. There is nothing in the current setup to encourage that. Then why does it do better? In a 5 branch setup, how different are these branches? Do they make the same errors or different errors?  (Minor) 2. I would like to see more ImageNet results, especially with more efficient models like SENet.   3. How does ONE-E compare to Sparsely Gated MoE [1]?  4. It was not mentioned how many low-level layers where shared.   [1] https://arxiv.org/pdf/1701.06538.pdf