To collect large scale annotated data, it is inevitable to introduce label noise, i.e., incorrect class labels. A major challenge is to develop robust deep learning models that achieve high test performance despite training set label noise. We introduce a novel approach that directly cleans labels in order to train a high quality model. Our method leverages statistical principles to correct data labels and has a theoretical guarantee of the correctness. In particular, we use a likelihood ratio test (LRT) to flip the labels of training data. We prove that our LRT label correction algorithm is guaranteed to flip the label so it is consistent with the true Bayesian optimal classifier with high probability. We incorporate our label correction algorithm into the training of deep neural networks and train models that achieve superior testing performance on multiple public datasets.
This paper proposes a new method for correcting training label noise, using a likelihood ratio test on the predicted probability of the classifier trained on the noisy labels, and then proposes an algorithm for iteratively cleaning training labels and retraining a model.  The paper also provides a theoretical guarantee for the probability of correctly re-labeling the training set, and provides empirical results showing that the proposed method significantly outperforms existing approaches for handling noisy training labels.
Overall, I think the empirical results appear very strong, but think this paper is below the acceptance threshold due to three factors, in ranked order:
- (1) The theoretical guarantee, which is positioned as a core contribution of the paper (and in fact claims it as "the first to correct labels with theoretical guarantees", which is not true), is based on assumptions that seem overly strong; these are somewhat relaxed in a "Remark", but this seems unproven and is a confusing presentation regardless.
- (2) The theoretical bound itself is somewhat vacuous as it contains several constant factors that seem very material to the bound, but totally opaque to the reader.
- (3) The experiments are very strong overall, which is a major plus for the paper; however, there are some questions about the hyperparameter tuning and some other points where more clarity could improve the strength of the empirical results
Regarding (1):
As a reader, my first natural reaction was to worry about circularity/degeneracy in the proposed method: basically, we are using the confidence (as a ratio of predicted cond. probabilities) of the model trained on the corrupted labels to correct those labels... if the labels are so corrupted that the model is also way off, then intuitively, this method should not work.  I was wondering about how this situation would be bounded / handled.
It turns out in Thm 1 that an incredibly strong assumption is made, namely that the model trained on the corrupted labels, f, is a linear function of the true model, with constants a, b known to some small degree of error epsilon (note that the theorem statement says that these constants are unknown- but it then assumes that \Delta, which is set based on a and b, is known up to \epsilon error).  This seems like an incredibly strong assumption- and no context / motivation is given about why it should be taken as reasonable.
Then, immediately after the Theorem, "Remark 2" states that this condition is not actually needed at all- but (a) then why not just strike it from Thm 1 statement, and (b) there does not seem to be any proof of this Remark in the appendix (where the proof of the main theorem itself is closer to a sketch than a standard proof...).
Regarding (2):
The bound produced in Thm 1 seems somewhat vacuous: letting \tau_{01} = \tau_{10}, then the probability of the label correction being erroneous is bounded by 8C(O(\epsilon))^\lambda.  This quantity is presumably in (0,1/2], so it's a small range to start... but it seems hard to get anything from this bound without some idea of what the constant factors (C, and those hidden in O(\epsilon)) and \lambda are.  In particular, as presented, it seems implausible that \epsilon- the error in specifying the \Delta threshold- gets that small, in which case these constants become very important to know!  Another way of phrasing this remark: many theoretical bounds have lots of unknown constant factors, but are ultimately just trying to expose some scaling with respect to one parameter, e.g. number of data points, and therefore the constants don't need to be known that well for the statement to have some value.  This doesn't exactly seem to be the case here- therefore it seems hard to extract something from this statement (even ignoring the strong assumptions it is predicated on).
Regarding (3):
Overall, I think the empirical performance reported in Table 2, and overall thoroughness of the ablation in Section 4, are major strong points for the paper- the performance is very impressive!  However I have a few questions, clarification of which would be very helpful in my mind:
- (i) A major issue that seems to be raised in the earlier sections is that there is a hyperparameter \Delta--the threshold for the likelihood ratio test--that everything depends on, and must be chosen empirically.  Table 5 shows that the effect of choosing it is not crazy, but also clearly not insignificant.  My question is: how is it chosen?  On the validation dataset?  And is this validation dataset also corrupted in the same way as the training dataset?  If not, that seems like a major whole in the setup.
- (ii) I also have a high level question for understanding: how is it possible for the various approaches to do so well with 0.6 and 0.8 noise level of uniform flipping?  In the p=0.8 noise model, for example, the probability of a data point getting flipped to *any individual wrong label* is *greater than that of it being the correct label*.  How is it possible to learn a model based on such a dataset?  Was there some kind of pre-training?  Was the validation set not corrupted?  I don't conceptually understand how the results shown are possible...?
Overall, I think points in (1) and (3) could be helped with additional clarification and contextualization, and possibly (2) as well.