Performing controlled experiments on noisy data is essential in thoroughly understanding deep learning across a spectrum of noise levels. Due to the lack of suitable datasets, previous research have only examined deep learning on controlled synthetic noise, and real-world noise has never been systematically studied in a controlled setting. To this end, this paper establishes a benchmark of realworld noisy labels at 10 controlled noise levels. As real-world noise possesses unique properties, to understand the difference, we conduct a large-scale study across a variety of noise levels and types, architectures, methods, and training settings. Our study shows that: (1) Deep Neural Networks (DNNs) generalize much better on real-world noise. (2) DNNs may not learn patterns first on real-world noisy data. (3) When networks are fine-tuned, ImageNet architectures generalize well on noisy data. (4) Real-world noise appears to be less harmful, yet it is more difficult for robust DNN methods to improve. (5) Robust learning methods that work well on synthetic noise may not work as well on real-world noise, and vice versa. We hope our benchmark, as well as our findings, will facilitate deep learning research on noisy data.
===========
Summary:
This paper introduces two real-world noisy datasets collected from google search based on two existing datasets: Mini-ImageNet and Stanford Cars. The author then conducted a series of experiments comparing 6 existing noisy label learning methods in two training settings: 1) from scratch, and 2) finetuning. Parameters were tuned when necessary. Based on the results, they made a few claims that challenges some of the previous believes in this field. 
===========
My major concerns:
Collection new data using google search then run existing methods has limited contribution. And the new data are related to noisy LABELS only, which does not cover any input noise such as low resolution, abnormal size, black/blank/carton background etc.
The web data searched are of moderate or rather small scales, which doesn’t seem to contribute much to the community. As mentioned in the paper, we already got some real-world datasets such Clothing-1M.
How the new datasets are collected, labeled and mixed are not clear. The numbers do not seem to add up. In text, the authors said they collected 94906/51687 images with annotations. However, in Table 1, there are 39000/8144 in datasets Red Mini-ImageNet/Red Standford Cars. The last paragraph on Page 3, how exactly the negative examples come from? How the “negative” is defined, since you got more than 1 nanotations for each image. “ Following the construction of synthetic datasets, we replace the training images in the original dataset”, refers to which original dataset? The Mini-ImageNet/Red Standford Cars or the web-searched? I don’t quite get it which dataset is creating here. Are the Red noise datasets a mixture of Mini-ImageNet/Red Standford Cars and web-search images, or are they only web-searched images.
The findings are questionable, due to the specific way how the web data are collected. Since the web datasets are created by similarity search based on a pool of seed images (5000), all the searched images are subject to the small number of seed images and the similarity algorithm of google search. Although some filtering was done to reduce test vs training duplication, the entire dataset can be treated as a duplicate of the 5000 seed images, which is roughly 10% of Mini-ImageNet -- the intrinsic noise rate is low. Having this in mind, some of the findings become no surprise at all, for example, 1) DNNs are robust to real-world noise, 2) performance does not decrease much as training progresses, 3) real-world noise is less harmful, and 4) DNN performance does not change much with different robust methods. These are all phenomena of low noise rates.