Sample-selection approaches, which attempt to pick up clean instances from the noisy training data set, have become one promising direction to robust learning from noisy labels. These methods all build on the memorization effect, which means deep networks learn easy patterns first and then gradually over-fit the training data set. In this paper, we show how to properly select instances so that the training process can benefit the most from the memorization effect is a hard problem. Specifically, memorization can heavily depend on many factors, e.g., data set and network architecture. Nonetheless, there still exists general patterns of how memorization can occur. These facts motivate us to exploit memorization by automated machine learning (AutoML) techniques. First, we design an expressive but compact search space based on observed general patterns. Then, we propose to use the natural gradient-based search algorithm to efficiently search through space. Finally, extensive experiments on both synthetic data sets and benchmark data sets demonstrate that the proposed method can not only be much efficient than existing AutoML algorithms but can also achieve much better performance than the state-of-the-art approaches for learning from noisy labels.
This paper focuses on the topic of learning from noisy -- or as they call it "corrupted" -- labels. Specifically this focuses on an approach where data selection -- ideally of cleaner/less noisy examples --  can help the learn model overcome data noise, akin to the approaches this builds upon (i.e, the Co-Teaching and MentorNet approaches). The specific idea here is to take an AutoML style approach to the problem  in particular to determine how many examples are selected in each mini-batch. The proposed method is based upon natural gradient based updates to the hparams (which was really the only feasible way to tackle this problem given the complex dependence on the hparams and a good choice). The experimental results using synthetic noise corruption are indicative of improved performance compared to the baseline techniques.
Overall while I thought the paper made for a very interesting read and showed some great promise I had some significant concerns as well.
On the plus side:
+ The empirical results on the simulated noisy data are quite positive/
+ The proposed method makes sense as does the search algorithm in the hparam space.
My main concerns with the work stem from the empirical study and choices made there. While I understand that other existing techniques like the Co-teaching and MentorNet approaches have used simulated noise to study the impact of performance of these robustness techniques, at some point I question their validity on real datasets. Noise patterns in real datasets hardly follow some set pattern and thus I hesitate to read much into results derived solely on synthetic datasets. Given that the goal of these techniques is to improve performance when training with real, noisy labeled data why not actually demonstrate performance on such benchmarks? For example, there are numerous datasets from domains like crowdsourcing that allow you to get "noisy" ratings for datapoints. Wouldn't a more compelling argument be derived by showing improved performance on such datasets?
Thus to summarize: I worry that the results derived solely on simulated noise may not be very indicative of performance in more realistic settings and would request the authors to consider providing evidence on more realistic datasets.
I also wanted to note that the paper exposition is lacking in some aspects and I needed to reread certain sections to make sure I understood them correctly. I think the paper would benefit from a good proofread not just from the grammar/spelling perspective (which there are multiple instances which could be improved) but also from the overall presentation and legibility perspective.
All this said: I want to clarify that this topic is not my research focus and hence I am uncertain as to how much findings on these simulated noise patterns carry over to real datasets and their associated noise patterns. If there is existing evidence indicating strong correlation, then perhaps my review may have varied.