It is challenging to train deep neural networks robustly on the industrial-level data, since labels of such data are heavily noisy, and their label generation processes are normally agnostic. To handle these issues, by using the memorization effects of deep neural networks, we may train deep neural networks on the whole dataset only the first few iterations. Then, we may employ early stopping or the small-loss trick to train them on selected instances. However, in such training procedures, deep neural networks inevitably memorize some noisy labels, which will degrade their generalization. In this paper, we propose a meta algorithm called Pumpout to overcome the problem of memorizing noisy labels. By using scaled stochastic gradient ascent, Pumpout actively squeezes out the negative effects of noisy labels from the training model, instead of passively forgetting these effects. We leverage Pumpout to upgrade two representative methods: MentorNet and Backward Correction. Empirical results on benchmark vision and text datasets demonstrate that Pumpout can significantly improve the robustness of representative methods.
A new method for defending against label noise during training of deep neural networks is presented. The main idea is to “forget” about wrongly labeled examples during training by an ascending step in the gradient direction. This is a meta-approach that can be combined with other methods for noise robustness; the detection of the noisy examples is specific of the base method utilised. Experimental results are promising.
In general, I find the idea original, and of potential practical use, but I believe the contribution of paper to be limited and not well supported by experiments. Moreover, I think that some claims made in this work are poorly justified.
== Method
The paper would greatly benefit from some theoretical backing of the proposed optimization scheme, even on a simplified scenario. An idea would be to prove that, given a dataset with noisy labels, PumpOut converges close to the best model (= the one learned without noise), for certain hyperparameters. I think this would be new and interesting. A result of similar fashion was proven in [A].
I found the following arguments not well or only heuristically supported:
* section 2: the scaling factor \gamma. Why using \gamma=1 is suboptimal? One could claim that as much as you want to memorize the true image-label patterns, you also want to forget the image-noise ones.
* Why MentorNet + PumpOut does not suffer from the selection bias effect of CoTraining? This is unclear to me
* The non-negative version of the BackwardCorrection, and its appeal to Kiryo et al 17 is interesting, but it sidesteps its justification. A loss that can be negative does not necessarily means that it not lower-bounded. In fact, for a minimization problem to be well-defined, all you need is a lower bounded objective. Then, adding the lower bound makes your loss non-negative. Notice that Patrini et al 17 did not state that BC is unbounded, but only that it can be negative. Can you show more that that -- maybe, at least experimentally?
The statement of Theorem 2 is trivial. In fact, no proof is given as it would be self-evident. Moreover, the Theorem is not used by PumpOut. Algorithm 3 uses a if-else conditional on the scale of the backward correction, without the max(0, .). I suggest to remove this part. I have notice later that the non-negative version of BC is used as a baseline in the experiment, but I think that is the only use.
Regarding the presentation, in section 3, I suggest to move the explanation of MentorNet and BackwardCorrection before their upgrade by PumpOut.
== Experiments
Table 1 can be removed as these are extremely common datasets.
The experimental results look very promising for applications. As a side effect of this analysis, I can also notice an improvement over BC given to the nnBC, which is nice per se. Although, I would have strengthen the empirics as follow.
* SET2 is only run on MNIST. Why not even on CIFAR10 which is used in SET1? Any future reader will wonder “did it work on CIFAR10?"
* A much harder instance of noise, for instance open set [B] or from a real dataset [Xiao et al 15] would have more clearly supported the use of PumpOut for real applications.
* Can the authors elaborate on “the choices of \beta and \gamma follows Kirkyo et al 2017” ? And how assuming their knowledge gives a fair comparison to BC which does not require them? I believe this is a critical point for the validity of the experiments.
Minor:
* “LRELU active function’ -> activation function. What is a LRELU? LeakyReLU?
[A] Malach, Eran, and Shai Shalev-Shwartz. "Decoupling" when to update" from" how to update"." Advances in Neural Information Processing Systems. 2017.
[B] Veit, Andreas, et al. "Learning From Noisy Large-Scale Datasets With Minimal Supervision." CVPR. 2017.