Deep neural networks are known to be annotation-hungry. Numerous efforts have been devoted to reduce the annotation cost when learning with deep networks. Two prominent directions include learning with noisy labels and semi-supervised learning by exploiting unlabeled data. In this work, we propose DivideMix, a novel framework for learning with noisy labels by leveraging semi-supervised learning techniques. In particular, DivideMix models the per-sample loss distribution with a mixture model to dynamically divide the training data into a labeled set with clean samples and an unlabeled set with noisy samples, and trains the model on both the labeled and unlabeled data in a semi-supervised manner. To avoid confirmation bias, we simultaneously train two diverged networks where each network uses the dataset division from the other network. During the semi-supervised training phase, we improve the MixMatch strategy by performing label co-refinement and label coguessing on labeled and unlabeled samples, respectively. Experiments on multiple benchmark datasets demonstrate substantial improvements over state-of-the-art methods. Source code to reproduce all results will be released.
This paper proposes a DivideMix framework for learning with noisy labels, where they first Co-Divide the training data into a labeled clean set and an unlabeled noisy set by modeling the per-sample loss distribution with GMM and using the small loss trick, then they exploit MixMatch to train the model on those labeled and unlabeled data in a semi-supervised manner. Experiments and comparisons with SOTA are provided, together with an ablation study.
Pros:
-The paper bridges the area of learning with noisy labels with semi-supervised learning and proposes an interesting method to learn with noisy labels in a semi-supervised manner. The treatment proceeds from analyzing good unsupervised loss functions, improving the MixMatch and implementing thorough experiments.
-The paper is clear and flows smoothly. The treatment is thorough, proceeding from designing algorithms, implementing experiments and an ablation study.
-The impact of the method is a clear asset. Most existing label noise works focus on certain noise models, but this paper is more general and proposes an interesting method to learn with noisy labels in a semi-supervised manner, and the experimental results are promising.
-The effort made on designing the confidence penalty for asymmetric noise is interesting.
Remarks:
-Sec 3.1: it seems that \tau is an important hyperparameter for dividing the noisy data into labeled and unlabeled sets, but in Sec 4 I only see that \tao is set to be 0.5 or 0.6 for CIFAR, what about other experiments? I’m also wondering if \tao needs some decay during training? For example, it may be 0.5 at early training epochs, but may be smaller at last, since deep NNs are gradually fitting noisy features during training.
-Algorithm: the proposed method needs to train two networks simultaneously. During each epoch, it firstly divides the noisy data by modeling the per-sample loss distribution with GMM, and then do MixMatch with label co-refinement and co-guessing. I’m a bit concerned about efficiency. So how about the computation time?
-Experiments: more details on experimental protocol may be needed: what kind of hyperparameter tuning was done? How many repeated runs? It would be helpful to report the means and standard deviations based on repeated samplings.
Overall take: this paper proposes a thorough treatment of learning with noisy labels in a semi-supervised manner, designing the algorithm and testing it empirically, which is an interesting and important contribution. My only concern is about the novelty since the small loss trick in label noise and the MixMatch approach in SSL are already explored by many recent studies, but to the best of my knowledge, this paper is the first to unify them to solve label noise problems.