Memorization in over-parameterized neural networks can severely hurt generalization in the presence of mislabeled examples. However, mislabeled examples are to hard avoid in extremely large datasets. We address this problem using the implicit regularization effect of stochastic gradient descent with large learning rates. By the loss statistics, we are able to identify mislabeled examples with remarkable success. Then, we discard the mislabeled examples on the fly and continue training with the rest of the examples. This leads to ON-THE-FLY DATA DENOISING (ODD), a simple yet effective algorithm that is robust to mislabeled examples, while introducing almost zero computational overhead. Empirical results demonstrate the effectiveness of ODD on several datasets containing artificial and real-world mislabeled examples.
Thanks for the rebuttal. But, I am still not very convinced with the proposed results. For CIFAR-100 (0%), you get about 0.2% gain, for ImageNet (0%), you get about 0.2% loss in top-5 accuracy, and for WebVision, you get about 0.3% gain. I am not sure whether you can call these as statistically significant gains. I believe such gain/loss can be obtained with many other tweaks, such as the learning rate scheduling, as the authors have done. 
I believe extensive testing the proposed method on many real noisy datasets, not the synthetically generated ones, and showing the consistent gains would much strengthen the paper. But, at the current version, the only such result is Table 5, which is, again, not very convincing to me. 
So, I still keep my rating. 
=======
Summary:
The authors propose a simple empirical method for cleaning the dataset for training. By using the implicit regularization property of SGD-based optimization method, the authors come up with a method of setting a threshold for the training loss statistics such that the examples that show losses above the threshold are regarded as noisy examples and are discarded. Their empirical results show that ODD (their method) can outperform other baselines when artificial random label noise is injected. They also show ablation studies on the hyperparameters and show the final result seems to be robust to those parameters. 
Pros:
- The method is very simple
- The empirical results, particularly on the synthetic noisy training data, seems to be encouraging.
- The ablation study argues that the method is robust to the hyperparameters, p, E, and h.
Cons:
- I think the results remains to be highly empirical. While it is interesting to see the division of the loss statistics in Figure 2, I am not very convinced about the real usage of the proposed method. The result in Table 5 shows that ODD can outperform ERM for real world datasets, but the improvement seems to be marginal. Moreover, the hyperparameter p was set to 30 for that experiment, but how did the authors choose that parameter? Clearly, if you choose wrong p, I think the performance will degrade, and it is not clear how you can choose p in real applications. The ablation studies are only with synthetic noisy label data, so I think the result is somewhat limited. 
- 
I think the paper shows interesting results, but my concern is that it seems to be quite empirical. The positive results are particularly on the synthetic data case.