Few-shot classification may involve differentiating data that belongs to a different level of labels granularity. Compounded by the fact that the number of available labeled examples are scarce in the novel classification set, relying solely on the loss function to implicitly guide the classifier to separate data based on its label might not be enough; few-shot classifier needs to be very biased to perform well. In this paper, we propose a model that incorporates a simple inductive bias: focusing on differences by building a dissimilar set of class representations. The model treats a class representation as a vector and removes its component that is shared among closely related class representatives. It does so through the combination of learned attention and vector orthogonalization. Our model works well on our newly introduced dataset − CIFAR-Hard − that contains different levels of labels granularity. It also substantially improved the performance on fine-grained classification dataset, CUB; whereas staying competitive on standard benchmarks such as mini-Imagenet, Omniglot, and few-shot dataset derived from CIFAR.
The stated contributions of the paper are: (1) a method for performing few-shot learning and (2) an approach for building harder few-shot learning datasets from existing datasets. The authors describe a model for creating a task-aware embedding for different novel sets (for different image classification settings) using a nonlinear self-attention-like mechanism applied to the centroid of the global embeddings for each class. The resulting embeddings are used per class with an additional attention layer applied on the embeddings from the other classes to identify closely-related classes and consider the part of the embedding orthogonal to the attention-weighted-average of these closely-related classes. They compare the accuracy of their model vs others in the 1-shot and 5-shot setting on various datasets, including a derived dataset from CIFAR which they call Hierarchical-CIFAR.
Overall, while we like the concepts/ideas and the problem is definitely important, we were not enthusiastic about the paper. First, we found the write up to be cryptic, involving very long unclear statements. It read as if the authors were writing for themselves and not for ICLR general audience. Beyond the writing style, we found the paper to have:
* Inadequate description of the model, including mathematical inaccuracies.
* Inadequate description of Hierarchical-CIFAR, motivation, and evaluation.
Description of model:
------------------------------
The manuscripts describe the presented approach as metric learning but make no use of a distance function in the different spaces they map to. Instead, the manuscript defines an inner product over embeddings to compute similarities.
The manuscript describe their “self-attention operation” as a dynamic set-to-set operation. While the usual definition of self-attention is permutation invariant, the definition presented here is not and thus cannot be accurately described as a mapping between sets. Specifically, in the usual presentation of self-attention the only sharing of information between different set elements is during the outer product of the key and query vectors. The use of a BLSTM between “neighboring elements” of the set of prototype vectors violates this assumption and induces a lack of permutation invariance. This makes the method sensitive to permutations of classes, which does not make sense for predicting unordered classes.
In sections 2.2.3 and 2.2.4 the material is presented twice but slightly differently. For example, the definition of $b_k$ inline before equation 3 differs from equation 6 later in the text.
Equation 7 is incorrect and should not exclude the current class from the denominator.
The description of how to classify new points after equation 6 is poorly explained. The description of what happens when $h_V$ is a “BLTSM” [sic (should be BLSTM)] is noninformative.
The manuscript describes two dimension sizes $H$ and $M$ but the definition of $attn$ requires that $H = M$.
Description of new dataset and evaluations:
------------------------------------------------------------
One of the stated contributions of the manuscript is a methodology to build harder few-shot learning datasets. Section 4.2 is the only place in the text that appears to address this point, but is unclear where either new finer-grained or coarser-grained labels are coming from (new manual annotation or otherwise). The manuscript “leave[s] out the detail of its construction for simplicity”, but it is unclear what is being done here in the first place.
The manuscript does not detail tuning competing methods on the new dataset and so it is unclear whether it is a fair comparison.
The manuscript presents evaluations without any discussion of differences in performance between datasets or the 1-shot/5-shot settings. For example, their method is significantly better on CUB on 1-shot but not so much on 5-shot, on the other hand it is not significantly better on 1-shot for H-CIFAR and CIFAR-HS but then becomes better than the rest with 5-shot.
Additional comments/corrections
---------------------------------------------
There were numerous typos and grammatical errors that were present in the manuscript that did not directly impact this evaluation but should be fixed in the future.