Learning with noisy labels is a common problem in supervised learning. Existing approaches require practitioners to specify noise rates, i.e., a set of parameters controlling the severity of label noises in the problem. The specifications are either assumed to be given or estimated using additional approaches. In this work, we introduce a technique to learn from noisy labels that does not require a priori specification of the noise rates. In particular, we introduce a new family of loss functions that we name as peer loss functions. Our approach then uses a standard empirical risk minimization (ERM) framework with peer loss functions. Peer loss functions associate each training sample with a certain form of "peer" samples, which evaluate a classifier' predictions jointly. We show that, under mild conditions, performing ERM with peer loss functions on the noisy dataset leads to the optimal or a near optimal classifier as if performing ERM over the clean training data, which we do not have access to. To our best knowledge, this is the first result on "learning with noisy labels without knowing noise rates" with theoretical guarantees. We pair our results with an extensive set of experiments, where we compare with state-of-the-art techniques of learning with noisy labels. Our results show that peer loss functions based method consistently outperforms the baseline benchmarks, as well as some recent new results. Peer loss provides a way to simplify model development when facing potentially noisy training labels, and can be promoted as a robust candidate loss function in such situations.
This paper proposed peer loss function for learning with noisy labels, combining two areas learning with noisy labels and peer prediction together. The novelty and the significance are both borderline (or below). There are 4 major issues I have found so far.
References: Looking at section 1.1 the related work, the references are a bit too old. While I am not sure about the area of peer prediction, in the area of learning with noisy labels (in a general sense), there were often 10 to 15 papers from every NeurIPS, ICML, ICLR and CVPR in recent years. The authors didn't survey the literature after 2016 at all... Nowadays most papers focus on sample selection/reweighting and label correction rather than loss correction in this area, but there are still many recent papers on designing more robust losses, see https://arxiv.org/abs/1805.07836 (NeurIPS 2018 spotlight), https://openreview.net/forum?id=rklB76EKPr and references therein. Note also that some label-noise related papers may not have the term label noise or noisy labels in the title, for example, https://openreview.net/forum?id=B1xWcj0qYm (ICLR 2019).
Motivation: The motivating claim "existing approaches require practitioners to specify noise rates" is wrong... Many loss correction methods can estimate the transition matrix T (which is indispensable in any loss correction) without knowing the noise rate, when there are anchor points or even no anchor points in the noisy training data. See https://arxiv.org/abs/1906.00189 (NeurIPS 2019) and references therein. See also the public comment posted by Nontawat when a special symmetric condition is assumed on the surrogate loss function.
Novelty: The paper introduced peer prediction, an area in computational economics and algorithmic game theory, to learning with noisy labels. This should be novel (to the best of my knowledge) and I like it! However, the obtained loss is very similar to the general loss correction approach, see https://arxiv.org/abs/1609.03683 (CVPR 2017 oral). This fact undermines the novelty of the paper, significantly. The authors should clarity the connection to and the difference from the loss correction approach.
Significance: The proposed method focuses on binary classification, otherwise the paper will be much more significant! Note that the backward and forward corrections can both be applied to multi-class classification. Moreover, similar to many theory papers, the experiments are too simple, where single-hidden-layer neural networks were trained on 10 UCI benchmark datasets. I have to say this may not be enough for ICLR that should be a more deep learning conference.