Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels
Deep learning with noisy labels is practically challenging, as the capacity of deep models is so high that they can totally memorize these noisy labels sooner or later during training. Nonetheless, recent studies on the memorization effects of deep neural networks show that they would first memorize training data of clean labels and then those of noisy labels. Therefore in this paper, we propose a new deep learning paradigm called "Co-teaching" for combating with noisy labels. Namely, we train two deep neural networks simultaneously, and let them teach each other given every mini-batch: firstly, each network feeds forward all data and selects some data of possibly clean labels; secondly, two networks communicate with each other what data in this mini-batch should be used for training; finally, each network back propagates the data selected by its peer network and updates itself. Empirical results on noisy versions of MNIST, CIFAR-10 and CIFAR-100 demonstrate that Co-teaching is much superior to the state-of-the-art methods in the robustness of trained deep models.
This paper proposes and empirically investigates a co-teaching learning strategy that makes use of two networks to cross-train data with noisy labels for robust learning. Experiments on MNIST, CIFAR10 and CIFAR100 seem to be supportive. The paper is well structured and well written, but technically I have following concerns.  1. Since the proposed co-teaching strategy is motivated by co-training, one can tell the similarity between the two. Typically, in a semi-supervised learning setup by co-training, the labeled and unlabeled data is known. It starts with the labeled data and improves two models based on the confidence of the unlabeled data. In the proposed co-teaching setup, however, whether a label is clean or noisy is unknown. So co-teaching starts with some "clean" labels and gradually builds two good models to deal with the noisy labels. Otherwise, co-teaching works very similar to co-training.  2. Since the quality of the labels is unknown, co-teaching has to estimate a few variables, for instance, the confidence of the labels and the noise rate. The former is done by so-called "small-loss", which is argued to give rise to "support vectors" and the latter is simply assumed to be known ($\tau$) in the experiments. This is one of my concerns. First of all, it is not clear how the estimate of $\tau$ would impact the performance of co-teaching. It would be helpful to show some results. Second, there is an implicit assumption behind co-teaching. That is the clean and noisy labels are uniformly distributed across mini-batches. In practice, both assumptions may not hold. Therefore, it is important to show the results with estimated $\tau$.  3. "dealing with extremely noisy labels" is claimed to be the focus of this paper. However, it is not clear what noisy rate is considered "extremely noisy". It may be a reasonable speculation that noise rate >50% is extremely noisy. Furthermore, why does co-teaching only work well for extremely noisy labels while not for slightly noisy labels? Wouldn't it be reasonable to show its behavior through a spectrum of noise rate, say, from clean, 10%, 20%, all the way to 80% (in a controlled manner, for instance, only on one dataset)?  4. Co-training has a very good theory backing it. The proposed co-teaching in this paper is basically empirical observations. It would be nice to come up with some similar analysis as co-training.  Overall, I find this paper clearly written. The idea is interesting but significance of the reported work is not overwhelming. There are some fundamental issues regarding the approach and the experimental results that need to be clarified.