Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels
Deep neural networks (DNNs) have achieved tremendous success in a variety of applications across many disciplines. Yet, their superior performance comes with the expensive cost of requiring correctly annotated large-scale datasets. Moreover, due to DNNs' rich capacity, errors in training labels can hamper performance. To combat this problem, mean absolute error (MAE) has recently been proposed as a noise-robust alternative to the commonly-used categorical cross entropy (CCE) loss. However, as we show in this paper, MAE can perform poorly with DNNs and challenging datasets. Here, we present a theoretically grounded set of noise-robust loss functions that can be seen as a generalization of MAE and CCE. Proposed loss functions can be readily applied with any existing DNN architecture and algorithm, while yielding good performance in a wide range of noisy label scenarios. We report results from experiments conducted with CIFAR-10, CIFAR-100 and FASHION-MNIST datasets and synthetically generated noisy labels.
log p(e y = e y n |x n , ✓)
p(e y = e y n |y = i)p(y = i|x n , ✓)), (1) where c represents number of classes, e y represents noisy labels, y represents the latent true labels and p(e y = e y n |y = i) is the (e y n , i)'th component of the confusion matrix. Usually, the real confusion matrix is unknown. Several methods have been proposed to estimate it [11,14,32,17,12]. Yet, accurate estimations can be hard to obtain. Even with the real confusion matrix, training with the above loss function might be suboptimal for DNNs. Assuming (1) a DNN with enough capacity to memorize the training set, and (2) a confusion matrix that is diagonally dominant, minimizing the cross entropy with confusion matrix is equivalent to minimizing the original CCE loss. This is because the right hand side of Eq. 1 is minimized when p(y = i|x n , ✓) = 1 for i = e y n and 0 otherwise, 8 n.
In the context of support vector machines, several theoretically motivated noise-robust loss functions like the ramp loss, the unhinged loss and the savage loss have been introduced [5,38,27]. More generally, Natarajan et al. [29] presented a way to modify any given surrogate loss function for binary classification to achieve noise-robustness. However, little attention is given to alternative noise robust loss functions for DNNs. Ghosh et al. [10,9] proved and empirically demonstrated that MAE is robust against noisy labels. This paper can be seen as an extension and generalization of their work.
Another popular approach attempts at cleaning up noisy labels. Veit et al. [39] suggested using a label cleaning network in parallel with a classification network to achieve more noise-robust prediction. However, their method requires a small set of clean labels. Alternatively, one could gradually replace noisy labels by neural network predictions [33,36]. Rather than using predictions for training, Northcutt et al. [31] offered to prune the correct samples based on softmax outputs. As we demonstrate below, this is similar to one of our approaches. Instead of pruning the dataset once, our algorithm iteratively prunes the dataset while training until convergence.
Other approaches include treating the true labels as a latent variable and the noisy labels as an observed variable so that EM-like algorithms can be used to learn true label distribution of the dataset [41,18,37]. Techniques to re-weight confident samples have also been proposed. Jiang et al. [16] used a LSTM network on top of a classification model to learn the optimal weights on each sample, while Ren, et al. [34] used a small clean dataset and put more weights on noisy samples which have gradients closer to that of the clean dataset. In the context of binary classification, Liu et al. [24] derived an optimal importance weighting scheme for noise-robust classification. Our method can also be viewed as re-weighting individual samples; instead of explicitly obtaining weights, we use the softmax outputs at each iteration as the weightings. Lastly, Azadi et al. [2] proposed a regularizer that encourages the model to select reliable samples for noise-robustness. Another method that uses knowledge distillation for noisy labels has also been proposed [23]. Both of these methods also require a smaller clean dataset to work.
3 Generalized Cross Entropy Loss for Noise-Robust Classifications
I acknowledge that I read the author's response, and I feel that the revised version of the manuscript will be even stronger. As a result, I am raising my score to an 8. This paper is definitely in the Top 50%.  -----  Summary:  This paper shows that using the negative Box-Cox transformation as a loss function ($L_q$ or Lq loss) generalizes both categorical cross-entropy (CCE) and mean absolute error (MAE) losses and offers an elegant way to trade off their respective virtues when training on noisy labels. The key insight comes from analyzing the loss function gradients: they are equivalent, except that CCE includes a term that implicitly assigns higher weights to incorrect predictions. This makes training with CCE faster than with MAE but also makes it more susceptible to overfitting label noise. Like CCE, the gradient of Lq loss yields a weighting term but with an exponent parameter that we can choose. When q=0, we get CCE, and when q=1, the weighting term disappears, which is equivalent to MAE. The paper shows that a variant of a known risk bound for MAE under uniform label noise applies to Lq loss as q approaches 1. Experimental results are noticeably strong: Lq consistently outperforms CCE and MAE both and is competitive with several alternative strong baselines.  My current feeling is that this paper is a pretty clear accept, but I look forward to reading the other reviewers' comments and authors' responses.  Quality:  This submission appears sound to me -- the bounds quoted from the Ghosh work are correct and the gradients for the CCE, MAE, and Lq losses are correct. The analyses of Lq loss (lines 147-153) and truncated Lq loss (lines 165-178) appear correct to me, but I can't guarantee their correctness. For what it's worth, I'd recommend including a more detailed proof in either an appendix or an extended version of the paper on arXiv.  The experiments provide convincing evidence that Lq loss is superior to CCE and MAE in the presence of uniform and class conditional label noise. Lq loss is competitive with Sukhbaatar and Fergus algorithm, even when it has access to the noise rate.  I do have two questions for the authors:  (1) How do these results apply to binary classification (presumably they do since binary classification is a special case)? However, Ghosh showed in an earlier paper [4] that there for binary classification, there are other noise robust losses, e.g., ramp loss, that work quite well and might provide a stronger baseline than MAE.  (2) Ghosh's results apply to more general label noise scenarios, including non-uniform noise. Does the analysis here also apply to this more challenging (and realistic) scenario?  The experiments do have one weakness: there has been a lot of research on noisy labels in the last few years, so I think that a comparison with additional baselines is warranted. Examples include [1][2][3]. Frankly, I'm not sure ANY baseline comparisions is strictly necessary since the paper's central claim is that Lq loss is superior to CCE and MAE -- not that it's superior to all noisy label learning techniques. However, the inclusion of the "Forward" method opens the door to the "why didn't you include this other baselinee" critique.  Clarity:  This paper is clearly written for the most part, though as noted, the theoretical analysis of Lq loss (lines 147-153, 165-178) would benefit from greater detail, even in a separate document.  The analysis of the gradients could use further clarification -- this sentence on lines 122-23 is particularly challenging to understand: "...less congruent with provided labels are implicitly weighed more in the gradient update." Staring at Equation 5, it's not immediately clear to me what this means.  The discussion of truncated Lq loss could benefit from further discourse. For example, what is the motivation behind the introduction of the threshold k in Equation 9? How can we intuitively understand what it's doing, and how does it connect to Equation 11?  Also, the details of the pruning step aren't entirely clear: since neural net training typically involves unconstrained SGD, how do we optimize v_i bounded between 0 and 1?  Originality:  This paper has a marginal amount of originality -- it leans heavily on the Ghosh and Ferrari/Yang papers, synthesizing their respective ideas (risk bounds for noise robust symmetric loss functions and Lq loss), rather than proposing entirely new ones. Nonetheless, I think it demonstrates innovtative thinking, especially when looking at the gradients of the loss functions and considering how they will impact the dynamics of neural net learning (wereas Ghosh focuses primarily on the statistical properties).  Significance:  I'm really uncertain about the significance of the paper. But my takeaway from the paper is "use Lq loss instead of CCE and MAE, in most cases." Given that Lq loss appears pretty easy to implement and differentiate, there seems to be no reason to replace CCE with Lq in any experiment where there may be nontrivial amounts of label noise.   References  [1] Reed, et al.  Training Deep Neural Networks on Noisy Labels with Bootstrapping. ICLR 2015 Workshop Track. [2] Natarajan, et al. Learning with Noisy Labels Nagarajan. ICML 2013. [3] Ma, et al. Dimensionality-Driven Learning with Noisy Labels. ICML 2018. [4] Ghosh, et al. Making Risk Minimization Tolerant to Label Noise. Neurocomputing 2015.