Recent semi-supervised learning methods have shown to achieve comparable results to their supervised counterparts while using only a small portion of labels in image classification tasks thanks to their regularization strategies. In this paper, we take a more direct approach for semi-supervised learning and propose learning to impute the labels of unlabeled samples such that a network achieves better generalization when it is trained on these labels. We pose the problem in a learning-to-learn formulation which can easily be incorporated to the state-of-the-art semi-supervised techniques and boost their performance especially when the labels are limited. We demonstrate that our method is applicable to both classification and regression problems including image classification and facial landmark detection tasks.
This paper uses a meta-learning approach to solve semi-supervised learning. The main idea is to simulate an SGD step on the loss of the meta-validation data and see how the model will perform if the pseudo-labels of unlabelled data are perturbed. Experiments on classification and regression problems show that the proposed method can improve over existing methods. The idea itself is intriguing but the derivation and some design choice are not very well-explained.
(1) The derivation from Eq.(3) to (4) is confusing. Note that in Eq.(3), the prediction \Phi_\theta also depends on \theta in addition to the pseudo-label z. When taking a step of SGD, the second term of Eq.(3) (with unlabelled data) will always be zero if both arguments of the loss (\Phi_\theta(x) and z_\theta(x)) change simultaneously. Eq.(4) somehow only considers the gradient of unsupervised loss, then the gradient would be zero because there is no incentive to deviate from the pseudo-label z. The pseudo-code does not help much. The update from \hat{\theta}^{t} to \hat{\theta}^{t+1} has the same issue: there is no incentive for \hat{\theta}^{t} to deviate because z is exactly produced by it.
(2) For classification problems, it is natural to use cross-entropy loss for the probability vector z. Are there any specific reasons for using Gumbel-softmax? In addition, using L2 loss for probability vectors (as mentioned in Appendix A) is known to be problematic as it may create exponentially many local minima (Auer et al, 1996).
(3) The recent work of Li et al. (2019) also considers iteratively improving pseudo-labels with meta-updates so it should be discussed and compared.
(4) Experiments
- What are the sizes of the meta-validation sets in the experiments?
- Error bars in the tables and Fig.2?
- The MM results in Table 2 are noticeably worse than the original results. For example, with 250 labeled data, MM achieved 11.08% in CIFAR-10 as reported in the original paper. (And 4000 labeled data can achieve 4.95%)
- It is said that option 2 is consistently better than option 1, which is not true for the MM baseline.
- 22500 training steps for Experiment 4 seems arbitrary. What are the candidates for the hyper-parameters?
Typos:
- In the first paragraph of Sec.2, one of the x and one of the y should be bold.
- Above Eq.(4), x^{U\in U} should be x^i \in U
- The transpose in Eq.(7) is not necessary
- It is said on page 6 that Fig.2 reports classification loss but the task is a regression problem.
Ref
- Auer, P., Herbster, M. and Warmuth, M.K., 1996. Exponentially many local minima for single neurons. In Advances in neural information processing systems (pp. 316-322).
- Li, X., Sun, Q., Liu, Y., Zheng, S., Chua, T.S. and Schiele, B., 2019. Learning to Self-Train for Semi-Supervised Few-Shot Classification. In Advances in neural information processing systems.