Since deep neural networks are over-parametrized, they may memorize noisy examples. We address such memorizing issue in the presence of annotation noise. From the fact that deep neural networks cannot generalize neighborhoods of the features acquired via memorization, we observe that noisy examples do not consistently incur small losses on the network under a certain perturbation. Based on this observation, we propose a novel training method called Learning with Ensemble Consensus (LEC) that prevents overfitting noisy examples by eliminating them using the consensus of an ensemble of perturbed networks. One of the proposed LECs, LTEC outperforms the current state-of-the-art methods on MNIST, CIFAR-10, and CIFAR-100 with efficient memory usage.
Summary: 
This paper proposes a general method for eliminating noisy labels in supervised learning based on the combination of two ideas: outputs of noisy examples are less robust under noise, and noisy labels are less likely to have a low loss. The authors then propose 3 concrete instantiations of the idea, and do a thorough empirical study (including ablations) across multiple architectures, datasets, noise types, and comparing to multiple related methods. The results show pretty convincingly that one of the new methods (LTEC) that uses past networks outputs to build an ensemble performs really well.
Caveats: 
1) I’m an emergency reviewer and had less time to do an in-depth review.
2) While my research is sufficiently close to review the paper, I’m not an expert on label noise specifically, so I cannot comment much on novelty and related work questions.
Comments:
* The readability of the paper could be dramatically improved by reporting results visually (eg bar-plots) and moving all tables into the appendix.
* The authors state that peak performance is valid because it *could* have been found using a validation set -- then why not just do that, and report this early-stopping performance everywhere, instead of always two numbers (peak and final)?
* Please make the perturbation properties in section 3.2 more precise: do you mean “there exists a threshold and perturbation such that for some (x, y)”? Or “For any threshold and any perturbation then for all (x, y) it holds...”? Or something in-between?
* For the “competing methods” paragraph, please cite the relevant papers in the main text, not only in the appendix.
* “over 4 runs” did you randomize the data noise in each run (and in the same way for each method?), or only the network initialisation?
* Table 5 is cool, but it raises the question: is 1.1epsilon the best, or would performance keep going up?
* Looking at the actual implementation of LTEC (Algorithm 4), I cannot resist the thought that M=infinity could work even better (at no extra cost): just maintain a monotonically shrinking set of samples?
Minor comments:
- Define the threshold symbol in section 3.2
- Define M in Algorithm 1
- Define (initial) \mathcal{P}_0 in Algorithm 4
- Fig 2: can you clarify what green is, does it correspond to “self-training”? 
- “label precision … does not decrease” -- well, not a lot, but it does decrease!
- Fig 1, Fig 2: set max y to 100
- Table 4: Include M=1 (which is self-training) for comparison