Despite deep neural networks have demonstrated extraordinary power in various applications, their superior performances are at expense of high storage and computational costs. Consequently, the acceleration and compression of neural networks have attracted much attention recently. Knowledge Transfer (KT), which aims at training a smaller student network by transferring knowledge from a larger teacher model, is one of the popular solutions. In this paper, we propose a novel knowledge transfer method by treating it as a distribution matching problem. Particularly, we match the distributions of neuron selectivity patterns between teacher and student networks. To achieve this goal, we devise a new KT loss function by minimizing the Maximum Mean Discrepancy (MMD) metric between these distributions. Combined with the original loss function, our method can significantly improve the performance of student networks. We validate the effectiveness of our method across several datasets, and further combine it with other KT methods to explore the best possible results. Last but not least, we fine-tune the model to other tasks such as object detection. The results are also encouraging, which confirm the transferability of the learned features.
This paper targets knowledge distillation of a large network to a smaller network. The approach is summarized by equations (3) and (4), which in short proposes that one should use the maximum-mean-discrepancy (MMD) of the network activations as a loss term for distillation.  
When considering CIFAR image classification tasks, it is shown that only when using a specific quadratic polynomial kernel (which as described in https://arxiv.org/pdf/1701.01036.pdf is tantamount of applying neural style transfer) the proposed approach is able to match the performance of the seminal paper of Hinton et al.  When embarking to imagenet, the proposed approach is only able to match the performance of standard knowledge distillation by adding the quadratic term (texture in neural style synthesis jargon). This is actually a sensible proposal. Yet, the claims about MMD as a way of explaining neural style transfer has appeared in the paper cited above, which the authors mention.
The idea of transferring from one domain to another using MMD as a regularizer appeared in https://arxiv.org/pdf/1605.06636.pdf by Long et al --- indeed equation (3) of this paper matches exactly equation (10) of Long et al. Note too that Long et al also discuss what kernels work well and which work poorly due to vanishing gradients and propose parametrised solutions. This is something this paper failed to do.
The two works cited above make me wonder about the novelty of the current paper.  In fact, this paper ends us being an application of the neural style transfer loss function to network distillation. As such this could be useful, if not already done by someone else previously.
I find that the paper is poorly written, with many typos, and lacks focus on a single concrete story. The CIFAR experiments fail to use KD+NST (ie the thing that works for imagenet - neural style transfer) and section 5.3 appears trivial in light of the cited works. For all these reasons, I am inclined to reject this paper.