TADAM: Task dependent adaptive metric for improved few-shot learning
Few-shot learning has become essential for producing models that generalize from few examples. In this work, we identify that metric scaling and metric task conditioning are important to improve the performance of few-shot algorithms. Our analysis reveals that simple metric scaling completely changes the nature of few-shot algorithm parameter updates. Metric scaling provides improvements up to 14% in accuracy for certain metrics on the mini-Imagenet 5-way 5-shot classification task. We further propose a simple and effective way of conditioning a learner on the task sample set, resulting in learning a task-dependent metric space. Moreover, we propose and empirically test a practical end-to-end optimization procedure based on auxiliary task co-training to learn a task-dependent metric space. The resulting few-shot learning model based on the task-dependent scaled metric achieves state of the art on mini-Imagenet. We confirm these results on another few-shot dataset that we introduce in this paper based on CIFAR100.
** After rebuttal   Thank you for the author response. I have read the rebuttal and will maintain my score. The submission and rebuttal motivate three orthogonal improvements (metric scaling, task-conditioning, auxiliary task co-training) to Prototypical Networks. The paper would be much better written as three distinct units that evaluate the three components separately and in more depth.  The mathematical analysis section of the paper consists of writing out the gradient of the objective function with the additional temperature parameter. Although this provides intuition **about the limiting cases of \alpha**, I would hesitate to call this a significantly "non-trivial" derivation. I also recommend that the authors provide empirical validation (e.g., plotting the contribution of \alpha-transformed gradients) for the non-limiting cases of \alpha in order to validate the intuition that varying \alpha in non-limiting cases trades off between "minimizing the overlap of the sample distributions or correcting cluster assignments sample-wise" (rather than just providing the final test performance as in Figure 3).  I also do not agree that the generalization to a learned temperature parameter is exactly analogous to the development of Prototypical Networks from Matching Networks: While the Prototypical Network is itself a slight modification to the Matching Network, it touches on the important tradeoff between parametric and non-parametric density estimation (e.g., see the section on mixture density estimation in the Prototypical Networks paper). I do not see substantial evidence for an analogous claim that the form of the gradient with the additional learnable temperature parameter "has implications for future work in few-shot learning (FSL) and metric learning more generally." The submission needs more argumentation in order to convince the reader that the derivation would "drive algorithm design and empirical studies" more than the provided study that varies \alpha.  Lastly, while it is nice to see evaluation on more datasets, the episodic training setup of CIFAR-100 is not sufficiently different from the few-shot miniImageNet dataset to be considered a major contribution to the community.  ** Before the rebuttal  This work proposes improvements to the prototypical network approach to few-shot classification introduced in [25]. The improvements are three-fold: (i) the addition of a learned temperature parameter in the computation of the softmax-normalized centroid distance; (ii) a modification to conditional normalization [16, 17] in which the conditioning is done as a function of the task-specific dataset as a whole; and (iii) an auxiliary prediction task implemented as an additional head in the architecture that outputs the logits for a 64-way classification task.  The method gives better performance on the standardized miniImageNet benchmark as well as on a few-shot split of CIFAR100 introduced by the authors.  QUALITY  The theoretical contribution of this paper is to write out the gradient of the objective function from [25] with the addition of the temperature scaling parameter in the softmax computation.  The limiting cases (zero and infinite temperature)  are discussed for intuition even though they do not occur in practice as I assume the temperature parameter is transformed to be positive finite (although this is not stated).  There are some flaws in quality throughout the paper:  Several technical claims / hypotheses made in the paper are not evidenced: "Neural network initialization and batch norm encourages this regime." "If Df is large, the network may have to work outside of its optimal regime to be able to scale down the feature representation." "We hypothesize that the importance of TEN is not uniformly distributed over the layers in the feature extractor. The lower layers closer to the similarity metric need to be conditioned more than the shallower layers closer to the image because pixel-level features extracted by shallow level layers are not task specific."  A conceptual point is made in the conclusion unnecessarily: it does not appear that the fact that "the scaled cosine similarity...does not belong to the class of Bregman divergences" is used anywhere in the paper, yet it is mentioned in the conclusion.  CLARITY   The paper is reasonably clear. Some points of are omitted: - Why not report results with temperature scaling and matching networks (instead of prototypical networks)? - What is the architecture of the TEN? What motivated this architecture (specifically, why use the centroids, which enable parameter sharing amongst the feature extractors, but still requires the TEN with additional parameters)? - What is the reason that the TEN "underperformed"? Was it overfitting?  The experimental comparison in and the discussion of Table 1 does not identify the differences in architectural complexity between the reported methods (i.e., [1, 14, 15] and the proposed work employ a ResNet architecture while the other methods employ a shallow convolutional network).  ORIGINALITY  The architecture and algorithm are a combination of previously proposed methods (see "SIGNIFICANCE" below).  The problem setup is not novel, although the authors apply the few-shot episode generation procedure of [30] to the CIFAR100 dataset.  SIGNIFICANCE  The work is incremental as a combination of previously proposed methods applied to prototypical networks for the task of few-shot classification. In particular: - "Metric scaling" is the addition of a learnable temperature parameter to the normalized distance computation in the regime of Matching Networks [30] or Prototypical Networks [25]. - "Task conditioning" makes use of a task-dataset-conditioning network to predict the scale and offset parameters of batch normalization layers as in [3, 16, 17].  - Auxiliary tasks are known to be beneficial to few-shot learning [*,**] and learning in general.  Moreover, I disagree with the argument (in the conclusion section) that the task sampling technique should be modified for improved performance on a few-shot task, as we should be hesitant about tuning the dataset (which is a feature of the problem) to an algorithm.  SPECIFIC COMMENTS  pg. 1: "...one aims to...learn a model that extracts information from a set of labeled examples (sample set)..." Introduce the terminology "support set" alongside "sample set". I believe the restriction to the labelled support/unlabelled query setting is not representative of recent works in few-shot learning; e.g., consider [***, ****], which deal with learning with unlabelled data in the support set.  pg. 1: It is strange to introduce few-shot learning with Ravi & Larochelle as the first citation, then to claim that the problem has subsequently been reframed by Ravi & Larochelle as meta-learning -- they are the same paper! This needs to be rewritten to correctly capture the nuanced difference between few-shot and meta-learning.  pg. 1: The claim that certain approaches to few-shot learning and meta-learning are "influential" and "central to the field" is subjective and   pg. 1: "a feature extractor (or more generally a learner)" A feature extractor is more general than a neural network with learned parameters, so this relationship needs to be reversed. Since you consider models with learned parameter, it would be sufficient to rewrite this as "a feature extractor with learned parameters."  line 57: "parameterized by \phi, mapping x to z, a representation space of dimension D_z" z is an element of the representation space, not the representation space itself.  line 59: "can directly be used to solve the few-shot learning classification problem by association" This needs a more thorough explanation and a more thorough description of the differences between [30] and [25]. In particular, the training criterion for [25] is discussed in lines 62-63 but [30]'s is not.  line 60: "proposed to introduce inductive bias" Framing the computation of the class centroid as an "inductive bias" is not useful in this context unless it is identified why it is a useful inductive bias.  line 61: "a unique feature representation" The mean of embeddings is not necessarily unique.  line 61: "for each class k" It is confusing to use k to index classes when above K have been used to count examples in each class.  line 77-8: "This is the case of Matching Networks [30], which use a Recurrent Neural Network (RNN) to accumulate information about a given task." The vanilla version of MNs does NOT use an RNN; only the "full-context embedding" version requires it.  line 108-109: "We observed that this improvement could be directly attributed to the interference of the different scaling of the metrics with the softmax." This needs an experimental result, and so likely does not belong in the "Model Description" section.  lines 224: "We re-implemented prototypical networks..." Do the experimental results remain the same when employing the authors' original code (https://github.com/jakesnell/prototypical-networks) with the addition of the temperature scaling parameter?  [*] Alonso, Héctor Martínez, and Barbara Plank. "When is multitask learning effective? Semantic sequence prediction under varying data conditions." arXiv preprint arXiv:1612.02251 (2016). [**] Rei, Marek. "Semi-supervised multitask learning for sequence labeling." arXiv preprint arXiv:1704.07156 (2017). [***] Finn, Chelsea, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. "One-shot visual imitation learning via meta-learning." In CoRL, 2017. [****] Metz, Luke, Niru Maheswaranathan, Brian Cheung, and Jascha Sohl-Dickstein. "Learning Unsupervised Learning Rules." arXiv preprint arXiv:1804.00222 (2018).