This paper introduces a new method to discover mislabeled training samples and to mitigate their impact on the training process of deep networks. At the heart of our algorithm lies the Area Under the Loss (AUL) statistic, which can be easily computed for each sample in the training set. We show that the AUL reliably reveals differences in training dynamics between (clean) samples that benefit from generalization and (mislabeled) samples that need to be "memorized". We demonstrate that the estimated AUL score conditioned on clean vs. noisy is approximately Gaussian distributed and can be well estimated with a simple Gaussian Mixture Model (GMM). The resulting GMM provides us with mixing coefficients that reveal the percentage of mislabeled samples in a data set as well as probability estimates that each individual training sample is mislabeled. We show that these probability estimates can be used to down-weight suspicious training samples and successfully alleviate the damaging impact of label noise. We demonstrate on the CIFAR10/100 datasets that our proposed approach is significantly more accurate and consistent across model architectures than all prior work.
In this paper, the authors proposed methodologies to identify mislabeled training examples. Specifically, the authors proposed to examine the training loss of each example to identify examples with high training losses. To mitigate the randomness in the training loss, the authors proposed an AUL (area under the loss) metric, which is less noisy that directly examining the loss. The authors proposed to use Gaussian mixture to approximate the AUL distribution, and then estimate the probability of mislabeling. The authors proposed to downweight training examples with high likelihood of being mislabeled. The proposal is examined in numerical studies.
While I agree the method has a good potential to be significant, in its current form, I have several concerns that prevent me from recommending acceptance.
1. How can we be sure that examples with large training loss are mislabeled examples, rather than examples that are inherently difficult to classify? For example, if the dataset consists of 99% bright images and 1% dark images, then it could be possible that dark image training examples have a higher loss. This question becomes more relevant when the model does not have sufficient complexity, or does not have the correct form of complexity.
2. If we cannot distinguish mislabeled examples with examples that are hard to classify, then it seems to me that a simple downweighting based on AUL could potential lead to the opposite outcome. Intuitively, we should upweight those difficult to classify examples to improve generalizability.
3. It is somewhat surprising to me that the AUL distribution is a nicely looking bimodal shape. Could it be due to the fact that the wrong labels are randomly generated with equal probabilities from other classes? What it we have a different error distribution, e.g., the label has p probability of being correct, and (1-p) probability of being k, where k is a constant?
4. Given the above concerns, I feel the numerical studies are insufficient. The authors may consider 1) including more noise distributions, 2) presenting the AUL distributions to check whether they are truly bimodal, and 3) trying more datasets and more model architectures.