Modern machine learning models are often trained on examples with noisy labels that hurt performance and are hard to identify. In this paper, we provide an empirical study showing that a simple k-nearest neighbor-based filtering approach on the logit layer of a preliminary model can remove mislabeled training data and produce more accurate models than some recently proposed methods. We also provide new statistical guarantees into its efficacy.
This paper proposes a k-NN method for identifying corrupted labels, and then applies this k-NN in the representation space of a deep neural net rather than the original feature space. Overall the paper is well written and the results look quite convincing
The theory appears to be important (if somewhat straightforward-looking) contributions of existing k-NN theory to the corrupted labels setting, based on the key quantity the authors defined as S_k, the minimum k-NN spread.
Since the theory highly depends on this quantity, the authors should after Definition 2 justify why they chose to base their results around S_k, and why intuitively it is the right quantity.
- I encourage another experiment where the label corruption is not completely at random, that is it depends on the values of x itself.  
- In particular I encourage a synthetic experiment where the authors look at corruptions with varying S_k (minimum k-NN spread) values, to empirically verify the their theory holds and this really is a meaningful quantity to consider in the label-corruption setting.
- Why don't the authors show the results for vanilla deep kNN trained on the full (noisy + clean) dataset in their experiments. This seems important to ascertain the benefits that might be attributed to simply switching to kNN.  Or is deep kNN generally worse that the original model trained on the full dataset?
- Why didn't the authors show the original model trained on the full dataset?
Is it because it always does worse than all the baselines considered in the paper?
I would expect it sometimes does much better than Control (eg. when noise rates are low), and this is the straightforward approach must practitioners would use.
- Why don't the authors present the accuracy of the k-NN method at identifying corrupted datapoints vs the other methods that aim to explicitly identify the corrupted datapoints?
In general, it seems the authors did not compare other filtering baselines, which would be more related to their method, for example:
Learning with Confident Examples: Rank Pruning for Robust Classification with Noisy Labels
Northcutt et al. (2017). https://arxiv.org/abs/1705.01936
- It would be valuable to the scientific community if the authors can comment on: 
Rolnick et al. (2018). Deep Learning is Robust to Massive Label Noise. https://arxiv.org/pdf/1705.10694.pdf  
- Some similar looking ideas have been proposed in: 
Gao et al. (2018). On the Resistance of Nearest Neighbor To Random Noisy Labels
https://pdfs.semanticscholar.org/4227/918020c15b719c415e93eb63d436583f1745.pdf
Parvin et al. (2010). A Modification on K-Nearest Neighbor Classifier.
https://globaljournals.org/GJCST_Volume10/7-A-Modification-on-K-Nearest-Neighbor-Classifier.pdf
so the authors should contrast their method/analysis against those papers.
- In Thm 1: "w.r.t. X" should be "w.r.t. x" (lower case)