Learning in the presence of label noise is a challenging yet important task, and it is crucial to design models that are robust when part of the dataset are mislabeled. In this paper, we discover that a new class of loss functions called the gambler's loss provides strong robustness to label noise across various levels of corruption. Training with this modified loss function reduces memorization of data points and is a simple yet effective method to improve robustness and generalization. Moreover, using this loss function allows us to derive an analytical early stopping criterion that accurately estimates when over-memorization begins to occur. Our overall approach achieves strong results and outperforming existing baselines.
Update after rebuttal:
The good:
The rebuttal and updated paper address many of my concerns. Most importantly, the updated paper demonstrates the three-stage phenomenon on Open Images and adds experiments on IMDB showing that the Gambler's loss with AES helps a lot. The LAES iteration introduced in the updated paper alleviates my concern about performance drop compared to the CT baseline at certain corruptions on CIFAR-10.
The bad:
- From Figure 12, it looks like the three-stage phenomenon doesn't hold on IMDB. Does AES provide additional benefit beyond the Gambler's loss on IMDB? This needs to be clarified with the way Figure 12 turned out.
- There is a serious missing citation [1] that should be included as a baseline. The proposed method in [1] is at least superficially similar to the Gambler's loss and also makes use of the fact that it is easier to fit clean labels than noisy labels. My apologies for not noticing this earlier.
Overall:
I would still suggest acceptance, because the three-stage phenomenon is an interesting find that the authors make good use of. In light of the missing citation, though, I cannot raise my score.
[1]: Zhilu Zhang, Mert R. Sabuncu. "Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels". NeurIPS 2018.
-----------------------------------------------------------------------
Summary:
This paper proposes a method to alleviate label noise. It opens with the observation of three distinct stages when training in the presence of label noise. Importantly, there is a ‘gap’ stage during which the network has not begun memorizing noisy labels and early stopping is ideal. The authors then observe that the Gambler’s loss (Ziyin et al., 2019) elongates the gap stage and propose an analytic early stopping (AES) criterion for identifying when to stop training.
The analysis of the AES criterion, e.g. in Figure 5, and the observation of a phase transition when tuning the o hyperparameter are quite interesting, and the latter observation is of practical value when using the AES criterion.
The AES criterion seems to be well-motivated, and the empirical evaluation of the Gambler’s loss with and without early stopping is good. The results are strong on MNIST but somewhat weak on CIFAR-10. Specifically, the improvements on CIFAR-10 only appear for large corruption rates (0.7+), and performance is lower than the baselines for other corruption rates. This is a worrying problem, because it calls into question the value of the method on larger problems. However, seeing as this is a distinct approach from the baselines and that it demonstrates some promise, I recommend borderline accept. The authors could raise my score by demonstrating more consistent gains on another larger-than-MNIST CV dataset or an NLP/speech dataset. Other points of concern that I have are listed below.
Major points:
At the top of page 3, the authors say that the idealized gap assumption “holds well for simple datasets such as MNIST and on datasets with very high corruption rate, where our method achieves best results, and less so on more complicated datasets such as CIFAR10”. The idealized gap assumption is behind the AES criterion, but Figure 5 suggests that the AES criterion works well on CIFAR-10, so what do the authors mean when they say the assumption doesn’t work as well on CIFAR-10? Is this just referring to the results?
Saying traditional label noise correction methods are “of no use when one is not aware of the existence of label noise” seems unfair. The FC method and others do not require foreknowledge of the corruption rate and do not harm performance in the absence of label noise, so they can also be said to automatically correct label noise.
“FC, however, requires knowing the whole transition matrix, and is outperformed significantly by our method.”
This is not quite true, because Patrini et al. propose an estimate of the transition matrix as part of the Forward correction. Did you use the estimated or true transition matrix for the FC method? It would be good to clarify this in the paper.
Minor points:
There are a few grammatical errors and typos in the paper:
“or explicit regularization, this is also what is suggested by Abiodun et al. (2018)” (run-on sentence)\
“CIFAR10” should be “CIFAR-10”