Over-parameterized deep neural networks trained by simple first-order methods are known to be able to fit any labeling of data. Such over-fitting ability hinders generalization when mislabeled training examples are present. On the other hand, simple regularization methods like early-stopping can often achieve highly nontrivial performance on clean test data in these scenarios, a phenomenon not theoretically understood. This paper proposes and analyzes two simple and intuitive regularization methods: (i) regularization by the distance between the network parameters to initialization, and (ii) adding a trainable auxiliary variable to the network output for each training example. Theoretically, we prove that gradient descent training with either of these two methods leads to a generalization guarantee on the clean data distribution despite being trained using noisy labels. Our generalization analysis relies on the connection between wide neural network and neural tangent kernel (NTK). The generalization bound is independent of the network size, and is comparable to the bound one can get when there is no label noise. Experimental results verify the effectiveness of these methods on noisily labeled datasets.
This paper proposes two regularization methods for learning on noisily labeled data: the first penalizes the distance w.r.t. Euclidean norm from an initial point and the second uses an additional auxiliary variable for each example to learn a noise. In the theoretical part, the paper shows that an original clean dataset can be learned from a noisily labeled dataset based on NTK-theory. Finally, the effectiveness of proposed regularization methods is well verified empirically for 2-layer NN, CNN, and ResNet on image classification tasks (MNIST, CIFAR-10).
Contributions:
- Propose two simple regularization techniques for learning from a noisily labeled dataset.
- Give generalization guarantees for these methods
Clarity:
The paper is well organized and easy to read.
Quality:
The work is of good quality and is technically sound.
Significance:
Since proposed methods are in some sense related to the early-stopping for the (stochastic) gradient descent, the developed theory is useful in understanding the generalization ability of over-parameterized neural networks falling into NTK-regime. Although an additive noise setting for the regression problem is rather common in statistical learning theory, an artificially flipped label setting for the classification problem may be new except for a few recent studies [Li+(2019)]. A result (Theorem 5.1) for the regression problem with the squared loss is not so surprising because the generalization error of gradient descent in a high-dimensional space (e.g., over-parameterized NNs) or an RKHS (i.e., infinite-dimensional model) has been well studied and the generalization error is composed of the (constant) variance and the distance between the model output and the true label. Thus, existing results of generalization error analyses for the regression are directly applied to bound the prediction error for clean labels. However, I basically like this paper and I think this paper makes a certain contribution to understanding the effect of over-parameterization.
A few questions:
- Usually, the regularization parameter goes to zero as the number of examples increases. Conversely, the regularization parameter in the proposed methods also increases. Could you explain why this difference happens?
- In classification tasks, the original problem setting is recovered by setting $p=lamba=0$. However, the generalization bound by Theorem 5.3 is still affected by $\lambda$. Is this theorem tight?
-----
Update: 
I thank the authors for the response. I have read the revised version of the paper and have confirmed an improvement of Theorem 5.2. I will keep my score. This paper is of good quality.