Learning from limited exemplars (few-shot learning) is a fundamental, unsolved problem that has been laboriously explored in the machine learning community. However, current few-shot learners are mostly supervised and rely heavily on a large amount of labeled examples. Unsupervised learning is a more natural procedure for cognitive mammals and has produced promising results in many machine learning tasks. In the current study, we develop a method to learn an unsupervised few-shot learner via self-supervised training (UFLST), which can effectively generalize to novel but related classes. The proposed model consists of two alternate processes, progressive clustering and episodic training. The former generates pseudo-labeled training examples for constructing episodic tasks; and the later trains the few-shot learner using the generated episodic tasks which further optimizes the feature representations of data. The two processes facilitate with each other, and eventually produce a high quality few-shot learner. Using the benchmark dataset Omniglot, we show that our model outperforms other unsupervised few-shot learning methods to a large extend and approaches to the performances of supervised methods. Using the benchmark dataset Market1501, we further demonstrate the feasibility of our model to a real-world application on person reidentification.
This paper considers the problem of learning an image representation for few-shot learning without using image labels during training. This is a well-motivated problem since (as the paper points out) learning such a representation using "episodes" of low-shot learning problems as examples may require a large amount of annotated data. The paper proposes an iterative algorithm which alternates between clustering the images using the current model and updating the model using the clusters as "pseudo-labels". This approach is not particularly elegant as there is no clear objective being optimized, but it may nevertheless be effective. One main claim of the paper is that the iterative nature of this process is key to the success of the algorithm. The choice of model is a multi-layer conv-net which is trained using SGD (Adam). The paper investigates both triplet (with and without hard negatives) and "prototype" losses for learning the model parameters. To find clusters, the paper adopts the DBSCAN algorithm using the Jaccard similarity of the k-reciprocal neighbour sets.
I am not aware of other papers that use self-supervised learning to obtain a representation which is specifically suitable for few-shot learning via nearest-neighbour classification. As is noted in the paper, the absence of supervision during training more closely resembles the scenario of few-shot learning in biological systems.
The design decisions are well motivated throughout the paper.
The appendices are high quality and make the paper much more complete. In particular: the method for choosing epsilon, the empirical study of the effect of epsilon and the discussion of the behaviour of the cluster sizes as training proceeds.
Principal concerns:
(1.1) The proposed approach is particularly similar to DeepCluster (Caron et al.). Besides the use of a different clustering algorithm, it seems that the main high-level difference is the use of "episodic training", in which the algorithm is trained to compare examples to a query example, rather than to classify single examples. I would have preferred to see a comparison to non-episodic training. (It might be necessary to train a linear classifier on top of the final feature representation rather than simply build a nearest-neighbour classifier, but still this is convex, cheap and even closed-form in the case of least-squares regression.)
(1.2) While the algorithm has been demonstrated on real images in the Market1501 dataset, it would have been much more convincing to see it demonstrated on a more widely-used dataset for few-shot learning such as Mini-ImageNet.
(1.3) There are several recent papers on unsupervised feature learning using self supervision, especially as an alternative to ImageNet-classification pre-training. There is no discussion of these approaches, yet they might perform better than the proposed algorithm, especially for tasks with real images. Some examples of such papers are:
- "Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks"
- "Representation Learning with Contrastive Predictive Coding"
- "Unsupervised representation learning by predicting image rotations"
The literature on using auto-encoders to learn feature representations is also relevant.
Issues with details in the paper:
(2.1) The DeepCluster paper observed that non-negligible accuracy could be achieved using a randomly-initialized conv-net (12% when chance is 0.1%). This enabled the use of clusters as pseudo-labels. Is a similar effect observed with your datasets and random initialization?
(2.2) For large datasets, the clustering algorithm might be prohibitively expensive? It would be useful to discuss the complexity of this algorithm.
(2.3) It would be interesting to see the effect of varying the frequency of the clustering during training (i.e. how many gradient steps are taken before updating the clustering).
(2.4) It is concerning that the accuracy drops sharply as \rho increases [6, 7, 8]*10^-5 in Table 4.
(2.5) It is stated that batch-normalization at the network output helps prevent over-fitting. Why? My intuition is that it would be more helpful for avoiding regions of the loss function which have a small gradient magnitude. What happens if you remove it?
(2.6) What is the test time procedure? Do you use the mean feature when there are k > 1 shots (even for the network trained with the hinge loss)? Do you L2-normalize the representation vectors?
(2.7) It seems potentially brittle to use hard negatives in the triplet loss with pseudo-labels? If the labels were wrong, then the hard negatives might not really be negatives. Nevertheless, this does not seem to be an issue, at least with these datasets.
(2.8) There are no error-bars anywhere in the paper.
Minor comments:
(3.1) Tables 3 and 4 would be easier to interpret if plotted on an axis.
(3.2) The word "concurrently" suggests that the clustering and the training are performed simultaneously. I would prefer "alternating".
(3.3) A grammatical review of the paper is required. For example in the abstract: "to a large extent (extend) and approaches the performance of (to the performances of) supervised methods".
(3.4) Remember to use \log and \max in latex to improve the appearance.