A significant issue in training deep neural networks to solve supervised learning tasks is the need for large numbers of labeled datapoints. The goal of semisupervised learning is to leverage ubiquitous unlabeled data, together with small quantities of labeled data, to achieve high task performance. Though substantial recent progress has been made in developing semi-supervised algorithms that are effective for comparatively small datasets, many of these techniques do not scale readily to the large (unlabeled) datasets characteristic of real-world applications. In this paper we introduce a novel approach to scalable semi-supervised learning, called Local Label Propagation (LLP). Extending ideas from recent work on unsupervised embedding learning, LLP first embeds datapoints, labeled and otherwise, in a common latent space using a deep neural network. It then propagates pseudolabels from known to unknown datapoints in a manner that depends on the local geometry of the embedding, taking into account both inter-point distance and local data density as a weighting on propagation likelihood. The parameters of the deep embedding are then trained to simultaneously maximize pseudolabel categorization performance as well as a metric of the clustering of datapoints within each psuedo-label group, iteratively alternating stages of network training and label propagation. We illustrate the utility of the LLP method on the ImageNet dataset, achieving results that outperform previous state-of-the-art scalable semi-supervised learning algorithms by large margins, consistently across a wide variety of training regimes. We also show that the feature representation learned with LLP transfers well to scene recognition in the Places 205 dataset.
The authors propose a local label propagation approach for large-scale semi-supervised learning. The approach learns a representation that tries to minimize a combination of the cross-entropy loss on the labeled data and a negative inner-product-based likelihood between the propagated pseudo-label and other examples with the same true label. The pseudo-labels on the unlabeled data are then calculated with a weighted k-NN scheme, where the weights take a heuristic correction of a soft similarity. Some further computational speedup is done with a memory cache described in an earlier work (Wu 2018b). Experimental results seem significantly superior to the competitors. The design choices are mostly justified with ablation studies.
The whole idea is interesting and the results are promising. From the current manuscript, my remaining concerns are
(1) How much contribution has readily been done by (Wu 2018a, b), and how much is the original design of the authors? From Section 3 (and without reading (Wu 2018a, b)), I cannot find a clear answer to this question. Currently it appears that the additional contribution over Wu's works is marginal.
(2) It is not clear to me how the proposed approach reaches the asserted efficiency over global label propagation approaches. In particular, each P(v_i)*Z in Equation (2) is O(N) to compute. Each w_j(v) in Equation (5) is O(K) to compute after getting all P(v_i)*Z, and then there are N (or at least N-M) such w_j(v) needed. So the total complexity is naively O(N (N-M) K). Even ignoring the K as a small constant, I cannot see how LLP is O(NM). Some running time profiling of LLP versus global LP might be helpful.
(3) For label propagation methods, it is important to understand whether the pseudo-labels are accurate and/or whether the methods might be mis-guided by the pseudo-labels. Is there any evidence on whether the pseudo-labels are accurate (absolutely, or with respect to the confidence)?
(4) For hyper-parameter selection, there is a "Learning rate is initialized to 0.03 and then dropped by a factor of 10 whenever validation performance saturates." But it is not clear how the validation set is formed, and what performance is measured. Is it a performance based on a labeled validation set (and if so, how large is the set) or unlabeled one?
I read the rebuttal. While it does not change my assessment, I thank the authors for clarifying some issues.