Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise
The growing importance of massive datasets used for deep learning makes robustness to label noise a critical property for classifiers to have. Sources of label noise include automatic labeling, non-expert labeling, and label corruption by data poisoning adversaries. Numerous previous works assume that no source of labels can be trusted. We relax this assumption and assume that a small subset of the training data is trusted. This enables substantial label corruption robustness performance gains. In addition, particularly severe label noise can be combated by using a set of trusted data with clean labels. We utilize trusted data by proposing a loss correction technique that utilizes trusted examples in a data-efficient manner to mitigate the effects of label noise on deep neural network classifiers. Across vision and natural language processing tasks, we experiment with various label noises at several strengths, and show that our method significantly outperforms existing methods.
The submission describes a training methodology when you are given a dataset partitioned into a trusted and untrusted dataset. In the abstract and introduction, the authors justify the need for such a method and give a brief overview of the related work. They call their method ‘Gold Loss Correction’ (GLC).  The algorithm describes approximating the probability of an untrusted label given an input by training some classifier with softmax output and then using this model alongside the trusted dataset to construct a matrix of corruption probabilities. They do this by approximating the corruption probabilities by simply measuring the likelihood a model will classify some class from the partition of the dataset that is without doubt of some true class. Ofcourse this is non-normalized, so they mention trying both a normalized (The corruption matrix becomes a confusion matrix) and the non-normalized version.  In either case they use the matrix to weight the output of a newly trained classifier to consider this likelihood. The results show that the normalized version performs slightly worse, and so GLC requires no normalization.  They compare this approach to other Training procedures that account for untrusted data, but only one other that utilizes a trusted set. The comparison to Forward Loss Correction has this inherent bias because it does not utilize the important information that is the foundation of this work. They attempt to remedy this by adjusting the Forward Loss Corrections corruption matrix with the identity on the trusted component of each dataset. This ended up not assisting, which they hypothesize is due to variance introduced in their setup. This specific differential doesn’t seem clear, and I feel further investigation into why the adjustment had little effect may shed light onto strengths and weaknesses of their algorithm.  The results show a strong advantage to using GLC over the other algorithms in most cases (except for one against distillation in a niche setting) as well as it being robust and data-efficient. Though it should be mentioned that GLC works on a baseline assumption that the trusted and untrusted labels are conditionally independent on x. They backup this assumption by saying most systems that induce noise usually do not know the true label which is valid, but it would be interesting to see the comparative results in cases where this assumption is broken in examples like worst-case adversarial settings.  The paper is well-written, and they describe both the related work and their procedures in detail. The concept underlying their innovative training methodology is not purely original because they are using the same foundation as in Patrini et al 2017, but then originally extend it to utilize trusted component of the dataset.