Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
Despite deep neural networks have demonstrated extraordinary power in various applications, their superior performances are at expense of high storage and computational costs. Consequently, the acceleration and compression of neural networks have attracted much attention recently. Knowledge Transfer (KT), which aims at training a smaller student network by transferring knowledge from a larger teacher model, is one of the popular solutions. In this paper, we propose a novel knowledge transfer method by treating it as a distribution matching problem. Particularly, we match the distributions of neuron selectivity patterns between teacher and student networks. To achieve this goal, we devise a new KT loss function by minimizing the Maximum Mean Discrepancy (MMD) metric between these distributions. Combined with the original loss function, our method can significantly improve the performance of student networks. We validate the effectiveness of our method across several datasets, and further combine it with other KT methods to explore the best possible results. Last but not least, we fine-tune the model to other tasks such as object detection. The results are also encouraging, which confirm the transferability of the learned features.
 This paper proposes a simple method for knowledge distillation. The teacher and student models are matched using MMD objectives, the author demonstrates different variants of matching kernels specializes to previously proposed variants of knowledge distillation. - The extensive evaluation suggests that the MMD with polynomial kernel provides better results than the previously proposed method. - It is interesting to see that MMD based transfer has more advantage on the object detection tasks. - Can the author provides more insights into the behavior of different kernels, for example visualizing, the gradient map might help us to understand why certain kernel works better than another one? - Did you consider translation invariance or other spatial properties when designing your kernels? In summary, this is an interesting paper with good empirical results. The technique being used generalization is quite straightforward, but the paper also includes a good amount of discussion on why the proposed approach could be better and I think that really helps the reader.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
Despite deep neural networks have demonstrated extraordinary power in various applications, their superior performances are at expense of high storage and computational costs. Consequently, the acceleration and compression of neural networks have attracted much attention recently. Knowledge Transfer (KT), which aims at training a smaller student network by transferring knowledge from a larger teacher model, is one of the popular solutions. In this paper, we propose a novel knowledge transfer method by treating it as a distribution matching problem. Particularly, we match the distributions of neuron selectivity patterns between teacher and student networks. To achieve this goal, we devise a new KT loss function by minimizing the Maximum Mean Discrepancy (MMD) metric between these distributions. Combined with the original loss function, our method can significantly improve the performance of student networks. We validate the effectiveness of our method across several datasets, and further combine it with other KT methods to explore the best possible results. Last but not least, we fine-tune the model to other tasks such as object detection. The results are also encouraging, which confirm the transferability of the learned features.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper proposes a simple method for knowledge distillation. The teacher and student models are matched using MMD objectives, the author demonstrates different variants of matching kernels specializes to previously proposed variants of knowledge distillation. - The extensive evaluation suggests that the MMD with polynomial kernel provides better results than the previously proposed method. - It is interesting to see that MMD based transfer has more advantage on the object detection tasks. - Can the author provides more insights into the behavior of different kernels, for example visualizing, the gradient map might help us to understand why certain kernel works better than another one? - Did you consider translation invariance or other spatial properties when designing your kernels? In summary, this is an interesting paper with good empirical results. The technique being used generalization is quite straightforward, but the paper also includes a good amount of discussion on why the proposed approach could be better and I think that really helps the reader.
assistant
 This paper proposes a simple method for knowledge distillation. The teacher and student models are matched using MMD objectives, the author demonstrates different variants of matching kernels specializes to previously proposed variants of knowledge distillation. - The extensive evaluation suggests that the MMD with polynomial kernel provides better results than the previously proposed method. - It is interesting to see that MMD based transfer has more advantage on the object detection tasks. - Can the author provides more insights into the behavior of different kernels, for example visualizing, the gradient map might help us to understand why certain kernel works better than another one? - Did you consider translation invariance or other spatial properties when designing your kernels? In summary, this is an interesting paper with good empirical results. The technique being used generalization is quite straightforward, but the paper also includes a good amount of discussion on why the proposed approach could be better and I think that really helps the reader.