ABSTRACT Semi-supervised learning, i.e. jointly learning from labeled an unlabeled samples, is an active research topic due to its key role on relaxing human supervision. In the context of image classification, recent advances to learn from unlabeled samples are mainly focused on consistency regularization methods that encourage invariant predictions for different perturbations of unlabeled samples. We, conversely, propose to learn from unlabeled data by generating soft pseudo-labels using the network predictions. We show that a naive pseudo-labeling overfits to incorrect pseudo-labels due to the so-called confirmation bias and demonstrate that mixup augmentation and setting a minimum number of labeled samples per mini-batch are effective regularization techniques for reducing it. The proposed approach achieves state-of-the-art results in CIFAR-10/100, SVHN, and Mini-ImageNet despite being much simpler than other methods. These results demonstrate that pseudo-labeling alone can outperform consistency regularization methods, while the opposite was supposed in previous work. Code will be made available.
Summary: This paper focuses on the semi-supervised learning problem, and proposes a way to improve previous pseudo-labeling methods.  In pseudo-labeling, there is an issue called confirmation bias, which accumulates the early errors of wrong pseudo labels.  By adding some simple tricks such as adding mixup augmentation and setting a minimum number of labeled samples per mini-batch, the confirmation bias is shown to be reduced, leading to an improvement in accuracy.  Experiments demonstrate that the additional tricks are meaningful and makes pseudo-labeling better than many baseline methods for semi-superivsed learning, including state-of-the-art consistency regularization methods.
Pros: This is an interesting paper with a clear motivation, which is to fix the so-called confirmation bias that appears in pseudo-labeling methods for semi-supervised learning.  Although the tricks introduced in the paper (mixup and changing the mini-batch selection rules) themselves are not novel, they make the proposed method simple.  It is also shown to be meaningful in reducing the confirmation bias in Table 1 and Figure 2, achieving the original goal of the paper.
Cons: The weakness of the paper is that the intuition or the motivation behind the design of the proposed method is not so clear.  Using mixup is justified by the reason that mixup gives better confidence calibration.  This is important for pseudo-labeling methods, because soft-label output predictions are used as pseudo labels.  On the other hand, however, it was not so obvious why a minimum number of labeled samples per mini-batch was considered.  Can we consider further extensions such as minimum number of labeled samples per mini-batch & per class?  (Perhaps the discussions about mixup and soft labels in the last paragraph of Section 3 should be more emphasized, for example in the last paragraph of the Introduction section.)
Related to the weakness above, it is hard to see how far the regularization effects of adding mixup and mini-batch sampling rules are contributing to add synergy to the pseudo-labeling methods.  This is partially answered with Figure 2, but it would make this easier to see if the experiments included stronger baselines, e.g., by adding the same regularization tricks to consistency regularization methods, perhaps in Table 3.
Finally, since future work on pseudo labels will follow this paper’s setup, hyperparameters such as lamba_A, lambda_H, and alpha should be chosen carefully instead of fixing them.
Other minor comments (that did not impact the score):
- In reference section, "Z. MaXiaoyu Tao" seems to combine two authors.
- Table 3 never appears in the text.  In Section 4.4, "The table" in the second sentence can be changed to "Table 3".
- "architecture plays and important role" --> "architecture plays an important role"
- In Table 2, "+" signs make it look like an equation.  I suggest using commas instead.
-  "ResNet arquitectures" --> "ResNet architectures"
~~~~~
Thank you for the response and for the additional discussions that were included in the updated paper.