Toward Robustness against Label Noise in Training Deep Discriminative Neural Networks
Collecting large training datasets, annotated with high-quality labels, is costly and time-consuming. This paper proposes a novel framework for training deep convolutional neural networks from noisy labeled datasets that can be obtained cheaply. The problem is formulated using an undirected graphical model that represents the relationship between noisy and clean labels, trained in a semisupervised setting. In our formulation, the inference over latent clean labels is tractable and is regularized during training using auxiliary sources of information. The proposed model is applied to the image labeling problem and is shown to be effective in labeling unseen images as well as reducing label noise in training on CIFAR-10 and MS COCO datasets.
The paper proposes a method for incorporating data with noisy labels into a discriminative learning framework (assuming some cleanly labeled data is also available), by formulating a CRF to relate the clean and noisy labels. The CRF sits on top of a neural network (e.g., CNN in their experiments), and all parameters (CRF and NN) are trained jointly via an EM approach (utilizing persistent contrastive divergence). Experiments are provided indicating the model outperforms prior methods on leveraging noisy labels to perform well on classification tasks, as well as inferring clean labels for noisily-labeled datapoints.
The technique is generally elegant: the graphical model is intuitive, capturing relationships between the data, clean/noisy labels, and latent "hidden" factors that can encode structure in the noisy labels (e.g., presence of a common attribute described by many of the noisy labels), while permitting an analytically tractable factorized form. The regularization approach (pre-training a simpler unconditional generative model on the cleanly labeled data to constrain the variational approximation during the early part of training) is also reasonable and appears effective.
The main downside of the technique is the use of persistent contrastive divergence during training, which (relative to pure SGD style approaches) adds overhead and significant complexity. In particular the need to maintain per-datapoint Markov chains could require a rather involved setup for truly large scale noisy datasets with hundreds of millions to billions of datapoints (such as might be obtained from web-based or user-interaction data).
The experiments are carefully constructed and informative. I was particularly happy to see the ablation study estimating the contribution of different aspects of the model; the finding that dropping the data <-> noisy-label link improves performance makes sense in retrospect, although wasn't something I was expecting. I would have been curious to see an additional experiment using the full COCO dataset as the "clean" data, but leveraging additional Flickr data to improve over the baseline (what would be the gain?).
The paper was quite readable, but could benefit from another pass to improve language clarity in a few places. For instance, the sentence beginning on line 30 took me a while to understand.
Overall, I think the paper is a valuable contribution addressing an important topic, so I recommend acceptance.