We study a simple generic framework to address the issue of bad training data; both bad labels in supervised problems, and bad samples in unsupervised ones. Our approach starts by fitting a model to the whole training dataset, but then iteratively improves it by alternating between (a) revisiting the training data to select samples with lowest current loss, and (b) re-training the model on only these selected samples. It can be applied to any existing model training setting which provides a loss measure for samples, and a way to refit on new ones. We show the merit of this approach in both theory and practice. We first prove statistical consistency, and linear convergence to the ground truth and global optimum, for two simpler model settings: mixed linear regression, and gaussian mixture models. We then demonstrate its success empirically in (a) saving the accuracy of existing deep image classifiers when there are errors in the labels of training images, and (b) improving the quality of samples generated by existing DC-GAN models, when it is given training data that contains a fraction of the images from a different and unintended dataset. The experimental results show significant improvement over the baseline methods that ignore the existence of bad labels/samples.
The authors propose an iterative method for discarding outlying training data: first, learn a model on the entire training dataset; second, identify the training examples that have high loss under the learned model; and then alternate between re-learning the model on the training examples that do not have high loss, and re-identifying the training examples with high loss under the new model. This method works for both supervised and unsupervised learning, and the authors show that in theory, their method has some convergence properties in the mixed linear regression and Gaussian mixture model settings. The authors also run some experiments on neural networks and datasets with synthetic noise to show the benefits of their proposed method.
The problem of noisy datasets is relevant to almost all machine learning problems in the real world, and the authors' method shows promise as a straightforward way to increase performance on such noisy data. However, my opinion is that the authors need to make a stronger case, theoretically and/or experimentally, for why their method should be preferred to other methods. Detailed comments follow.
== Experiments ==
1) No comparisons are provided to other outlier detector methods (e.g., based on nearest neighbors, distance to centroid, influence functions, etc.) or techniques that also purport to deal with noisy labels (e.g., by modifying the learning algorithm or loss function). While there are too many existing methods to expect the authors to benchmark against all of them, it's important to at least have a couple of representative comparisons. 
2) It'd be nice to have an ablative analysis to tease out the factors behind the gain in accuracy. For example, is the iteration important, or would a single pass suffice? How robust is the algorithm to tau, the fraction of data to discard? (The authors do test initializing randomly vs. initializing on the full dataset.) 
3) The systematic label noise scenario seems to favor the authors' method (though the authors claim that it is a harder scenario than random label noise). It'd be helpful to see if the method works against random noise.
== Theory ==
4) The assumptions seem very restrictive. For example, for mixed linear regression (section 4), the features of all examples are assumed to be drawn i.i.d. from an isotropic Gaussian (so even the bad samples are drawn from the same distribution as the good samples; and all features are independent). To my knowledge, this is not a "standard and widely studied" assumption. For the Gaussian mixture model (section 5), a similar isotropic Gaussian assumption is made for each mixture. 
5) Beyond the independence assumptions mentioned above, the initialization results make additional assumptions on the "bad" data (e.g., average distance of the good vs. bad parameters) that I found hard to parse. How strong are these assumptions? Do they hold on real datasets?
6) The convergence results (Theorems 1 and 3) have a constant term sigma in them. This is surprising and seems to me to considerably weaken the result -- one would expect that the dependence on sigma will decrease with n.
I think either a strong experimental or strong theoretical section would be sufficient for me to recommend acceptance. However, the paper currently shows potentially interesting experimental/theoretical results but does not do a comprehensive job of either side.