Distillation is a method to transfer knowledge from one model to another and often achieves higher accuracy with the same capacity. In this paper, we aim to provide a theoretical understanding on what mainly helps with the distillation. Our answer is "early stopping". Assuming that the teacher network is overparameterized, we argue that the teacher network is essentially harvesting dark knowledge from the data via early stopping. This can be justified by a new concept, Anisotropic Information Retrieval (AIR), which means that the neural network tends to fit the informative information first and the non-informative information (including noise) later. Motivated by the recent development on theoretically analyzing overparameterized neural networks, we can characterize AIR by the eigenspace of the Neural Tangent Kernel(NTK). AIR facilities a new understanding of distillation. With that, we further utilize distillation to refine noisy labels. We propose a self-distillation algorithm to sequentially distill knowledge from the network in the previous training epoch to avoid memorizing the wrong labels. We also demonstrate, both theoretically and empirically, that self-distillation can benefit from more than just early stopping. Theoretically, we prove convergence of the proposed algorithm to the ground truth labels for randomly initialized overparameterized neural networks in terms of 2 distance, while the previous result was on convergence in 0-1 loss. The theoretical result ensures the learned neural network enjoy a margin on the training data which leads to better generalization. Empirically, we achieve better testing accuracy and entirely avoid early stopping which makes the algorithm more user-friendly.
This paper proposes a new perspective on understanding knowledge distillation as a transfer of information defined with respect to the neural tangent kernel. Additionally, a new framework for learning the classifier on a noisy labeled dataset is proposed based on the knowledge transfer framework. 
Overall, I think the paper lacks justification (and explanation) for its main statement on how knowledge distillation is related to the early stopping of the teacher network. Especially, it is confusing since Section 2 and 3 make different statements. Specifically, Section 2 shows that early stopping "helps" knowledge distillation while Section 3 shows that knowledge distillation can "replace" early stopping. The former observation implies that early stopping is complementary to knowledge distillation, while the latter implies otherwise.
Furthermore, Section 2 mainly explains why the eigenspaces associated with the largest eigenvalue of the neural tangent kernel is "informative information". However, there is no elaboration on how the knowledge distillation process leads to the transfer of such information, i.e., there is no connection between the neural tangent kernel and the knowledge distillation process. Although Figure 1. suggests that early stopping indeed improves the knowledge distillation process, they are not enough to support the statement convincingly enough. 
Without proper support on the main statement of this paper, the paper looses much of its claimed contributions. The label refinery algorithm for the noisy labeled dataset is interesting, but it is not evaluated thoroughly enough to demonstrate its superiority over existing algorithms. It also does not have much originality when compared to similar algorithms [1, 2]. Bagherinezhad  et al., [1] also tried to remove "noisy supervisions" that were generated by harsh augmentation on images. Han et al., [2] and Li et al., [3] also use distillation-like processes to learning noisy datasets.
[1] Label Refinery: Improving ImageNet Classification through Label Progression, Bagherinezhad  et al., 2018
[2] Co-teaching: Robust Training of Deep NeuralNetworks with Extremely Noisy Labels, Han et al., 2018 
[3] Learning from Noisy Labels with Distillation, Li et al., 2017