summaries: The paper addresses the problem of multi-domain few-shot image classification (where unseen classes and examples come from diverse data sources), and proposes a Universal Representation Transformer (URT) layer, which learns to transform a universal representation into task-adapted representations. The method proposed builds on top of SUR [Dvornik et al 2020], where a universal representation is extracted from the outputs of a collection of pre-trained and domain-specific backbones and a selection procedure infers how to weight each backbone for a given task at hand. While SUR inferred those weights by optimising a loss on the support set (the few examples provided in a task), the authors in this paper introduce an attention-based layer (inspired by Vaswani et al Transformer) that learns to weight the appropriate backbones for each task. This layer has the main advantage that it can be learned across few-shot tasks from many domains so it can support transfer across these tasks. Few-shot learning on meta-dataset is challenging due to the domain gap between train and validation. In order to bridge this gap, the authors present a model that learns to combine domain-specific representations to generalize to new domains. This combination is done with a transformer model that pays attention to the features extracted from domain-specific backbones. The authors demonstrate empirically that their model attains comparable performance to previous state-of-the-art at higher efficiency and include ablation results to test their model components. The paper presents a method for tackling multi-domain few-shot image classification problem where it obtains a task-adapted representation by weighing representations from pretrained domain-specific backbones according to the support set at hand. The desirable property of this framework is that the model can leverage information from other domains to make predictions. The effectiveness of Universal Representations have been discussed in the past work - SUR [1], and this work builds on top of it and introduces a learnable component (self-attention), and showed the improvement both quantitatively and qualitatively.
abstractText: Few-shot classification aims to recognize unseen classes when presented with only a small number of samples. We consider the problem of multi-domain few-shot image classification, where unseen classes and examples come from diverse data sources. This problem has seen growing interest and has inspired the development of benchmarks such as Meta-Dataset. A key challenge in this multi-domain setting is to effectively integrate the feature representations from the diverse set of training domains. Here, we propose a Universal Representation Transformer (URT) layer, that meta-learns to leverage universal features for few-shot classification by dynamically re-weighting and composing the most appropriate domain-specific representations. In experiments, we show that URT sets a new state-of-the-art result on Meta-Dataset. Specifically, it achieves top-performance on the highest number of data sources compared to competing methods. We analyze variants of URT and present a visualization of the attention score heatmaps that sheds light on how the model performs cross-domain generalization. Our code is available at https://github.com/liulu112601/URT.
authors.name: Lu Liu
authors.name: William Hamilton
authors.name: Guodong Long
authors.name: Jing Jiang
authors.name: Hugo Larochelle
references.authors: Ferran Alet Tomás Lozano-Pérez Leslie P Kaelbling
references.title: Modular meta-learning
references.venue: In Conference on Robot Learning (CoRL),
references.authors: Peyman Bateni Raghav Goyal Vaden Masrani Frank Wood Leonid Sigal
references.title: Improved few-shot visual classification
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Hakan Bilen Andrea Vedaldi
references.title: Universal representations: The missing link between faces, text, planktons, and cat breeds
references.venue: arXiv preprint arXiv:1701.07275,
references.authors: Wei-Yu Chen Yen-Cheng Liu Zsolt Liu Yu-Chiang Frank Wang Jia-Bin Huang
references.title: A closer look at few-shot classification
references.venue: In International Conference on Learning Representations (ICLR),
references.authors: Yinbo Chen Xiaolong Wang Zhuang Liu Huijuan Xu Trevor Darrell
references.title: A new meta-baseline for few-shot learning
references.venue: arXiv preprint arXiv:2003.04390,
references.authors: Guneet Singh Dhillon Pratik Chaudhari Avinash Ravichandran Stefano Soatto
references.title: A baseline for few-shot image classification
references.venue: In International Conference on Learning Representations (ICLR),
references.authors: Nikita Dvornik Cordelia Schmid Julien Mairal
references.title: Selecting relevant features from a universal representation for few-shot classification
references.venue: arXiv preprint arXiv:2003.09338,
references.authors: Chelsea Finn Pieter Abbeel Sergey Levine
references.title: Model-agnostic meta-learning for fast adaptation of deep networks
references.venue: In The International Conference on Machine Learning (ICML),
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep residual learning for image recognition
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
references.authors: Alex Krizhevsky Geoffrey Hinton
references.title: Learning multiple layers of features from tiny images
references.venue: Technical report, University of Toronto,
references.authors: Brenden M Lake Ruslan Salakhutdinov Joshua B Tenenbaum
references.title: Human-level concept learning through probabilistic program induction
references.authors: Yann LeCun Léon Bottou Yoshua Bengio Patrick Haffner
references.title: Gradient-based learning applied to document recognition
references.venue: Proceedings of the IEEE,
references.authors: Hongyang Li David Eigen Samuel Dodge Matthew Zeiler Xiaogang Wang
references.title: Finding taskrelevant features for few-shot learning by category traversal
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
references.authors: Zhouhan Lin Minwei Feng Cicero Nogueira dos Santos Mo Yu Bing Xiang Bowen Zhou Yoshua Bengio
references.title: A structured self-attentive sentence embedding
references.venue: arXiv preprint arXiv:1703.03130,
references.authors: Lu Liu Tianyi Zhou Guodong Long Jing Jiang Lina Yao Chengqi Zhang
references.title: Prototype propagation networks (PPN) for weakly-supervised few-shot learning on category graph
references.venue: In International Joint Conferences on Artificial Intelligence (IJCAI),
references.authors: Lu Liu Tianyi Zhou Guodong Long Jing Jiang Chengqi Zhang
references.title: Learning to propagate for graph meta-learning
references.venue: In Neural Information Processing Systems (NeurIPS),
references.authors: Lu Liu Tianyi Zhou Guodong Long Jing Jiang Chengqi Zhang
references.title: Attribute propagation network for graph zero-shot learning
references.venue: In AAAI Conference on Artificial Intelligence (AAAI),
references.authors: Yang Liu Chengjie Sun Lei Lin Xiaolong Wang
references.title: Learning natural language inference using bidirectional lstm model and inner-attention
references.venue: arXiv preprint arXiv:1605.09090,
references.authors: Boris Oreshkin Pau Rodrı́guez López Alexandre Lacoste
references.title: Tadam: Task dependent adaptive metric for improved few-shot learning
references.venue: In The Conference on Neural Information Processing Systems (NeurIPS),
references.authors: Ethan Perez Florian Strub Harm de Vries Vincent Dumoulin Aaron C. Courville
references.title: Film: Visual reasoning with a general conditioning layer
references.venue: In AAAI Conference on Artificial Intelligence (AAAI),
references.authors: Sylvestre-Alvise Rebuffi Hakan Bilen Andrea Vedaldi
references.title: Efficient parametrization of multidomain deep neural networks
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
references.authors: Mengye Ren Eleni Triantafillou Sachin Ravi Jake Snell Kevin Swersky Joshua B Tenenbaum Hugo Larochelle Richard S Zemel
references.title: Meta-learning for semi-supervised few-shot classification
references.venue: In International Conference on Learning Representations (ICLR),
references.authors: James Requeima Jonathan Gordon John Bronskill Sebastian Nowozin Richard E Turner
references.title: Fast and flexible multi-task classification using conditional neural adaptive processes
references.venue: In The Conference on Neural Information Processing Systems (NeurIPS),
references.authors: Tonmoy Saikia Thomas Brox Cordelia Schmid
references.title: Optimized generic feature learning for fewshot classification across domains
references.venue: arXiv preprint arXiv:2001.07926,
references.authors: Jake Snell Kevin Swersky Richard Zemel
references.title: Prototypical networks for few-shot learning
references.venue: In The Conference on Neural Information Processing Systems (NeurIPS),
references.authors: Eleni Triantafillou Tyler Zhu Vincent Dumoulin Pascal Lamblin Utku Evci Kelvin Xu Ross Goroshin Carles Gelada Kevin Swersky Pierre-Antoine Manzagol
references.title: Meta-dataset: A dataset of datasets for learning to learn from few examples
references.venue: In International Conference on Learning Representations (ICLR),
references.authors: Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Łukasz Kaiser Illia Polosukhin
references.title: Attention is all you need
references.venue: In The Conference on Neural Information Processing Systems (NeurIPS),
references.authors: Oriol Vinyals Charles Blundell Tim Lillicrap Daan Wierstra
references.title: Matching networks for one shot learning
references.venue: In The Conference on Neural Information Processing Systems (NeurIPS),
references.authors: 2016. Han-Jia Ye Hexiang Hu De-Chuan Zhan Fei Sha
references.title: Few-shot learning via embedding adaptation
references.authors: 2020. Sung Whan Yoon Jun Seo Jaekyun Moon
references.title: Tapnet: Neural network augmented with task
references.authors: 2020 Triantafillou et al 2020 Dhillon et al Saikia et al 2020). Specifically Saikia
references.title: 2020) have shown that competitive performance can be reached using a strong hyper-parameter optimization method applied on a carefully designed validation metric appropriate for few-shot learning. C STRUCTURES AND TRAINING STRATEGY OF BACKBONES
sections.heading: 1 INTRODUCTION
sections.text: Learning tasks from small data remains a challenge for machine learning systems, which show a noticeable gap compared to the ability of humans to understand new concepts from few examples. A promising direction to address this challenge is developing methods that are capable of performing transfer learning across the collective data of many tasks. Since machine learning systems generally improve with the availability of more data, a natural assumption is that few-shot learning systems should benefit from leveraging data across many different tasks and domains—even if each individual task has limited training data available.
  This research direction is well captured by the problem of multi-domain few-shot classification. In this setting, training and test data spans a number of different domains, each represented by a different source dataset. A successful approach in this multi-domain setting must not only address the regular challenge of few-shot classification—i.e., the challenge of having only a handful of examples per class. It must also discover how to leverage (or ignore) what is learned from different domains, achieving generalization and avoiding cross-domain interference.
  Recently, Triantafillou et al. (2020) proposed a benchmark for multi-domain few-shot classification, Meta-Dataset, and highlighted some of the challenges that current methods face when training data is heterogeneous. Crucially, they found that methods which trained on all available domains would normally obtain improved performance on some domains at the expense of others. Following on their work, progress has been made, which includes the design of adapted hyper-parameter optimization strategies (Saikia et al., 2020) and more flexible meta-learning algorithms (Requeima et al., 2019). Most notable is SUR (Selecting Universal Representation) (Dvornik et al., 2020), a method that relies on a so-called universal representation, extracting from a collection of pre-trained and domain-specific neural network backbones. SUR prescribes a hand-crafted feature-selection procedure to infer how to weight each backbone for each task at hand, and produces an adapted representation for each task. This was shown to lead to some of the best performances on Meta-Dataset. ∗This work was done while Lu Liu was a research intern with Mila. †Canada CIFAR AI Chair
  In SUR, the classification procedure for each task is fixed and not learned. Thus, except for the underlying universal representation, there is no transfer learning performed with regards to how classification rules are inferred across tasks and domains. Yet, cross-domain generalization might be beneficial in that area as well, in particular when tasks have only few examples per class.
  Present work. To explore this question, we propose a Universal Representation Transformer (URT) layer, which can effectively learn to transform a universal representation into task-adapted representations. The URT layer is inspired from Transformer (Vaswani et al., 2017) and uses an attention mechanism to learn to retrieve or blend the appropriate backbones to use for each task. By training this layer across few-shot tasks from many domains, it can support transfer across these tasks.
  We show that our URT layer on top of a universal representation’s pre-trained backbones sets a new state-of-the-art performance on Meta-Dataset. It succeeds at outperforming SUR on 4 dataset sources without impairing accuracy on the others. This leads to top performance on 7 dataset sources when comparing to a set of competing methods. To interpret the strategy that URT learns to weigh the backbones from different domains, we visualize the attention scores for both seen and unseen domains and find that our model generates meaningful weights for the pre-trained domains. A comprehensive analysis on variants and ablations of the URT layer is provided to show the importance of various components of URT, notably the number of attention heads.
sections.heading: 2 FEW-SHOT CLASSIFICATION
sections.heading: 2.1 PROBLEM SETTING
sections.text: In this section, we will introduce the problem setting for few-shot classification and the formulation of meta-learning for few-shot classification. Few-shot classification aims to classify samples where only few examples are available for each class. We describe a few-shot learning classification task as the pair of examples, comprising of a support set S to define the classification task and the query set Q of samples to be classified.
  Meta-learning is a technique that aims to model the problem of few-shot classification as learning to learn from instances of few-shot classification tasks. The most popular way to train a meta-learning model is with episodic training. Here, tasks T = (Q,S) are sampled from a larger dataset by taking subsets of the dataset to build a support set S and a query set Q for the task. A common approach is to sample N -way-K-shot tasks, each time selecting a random subset of N classes from the original dataset and choosing only K examples for each class to add to the support set S.
  The meta-learning problem can then be formulated by the following optimization:
  min Θ
  E(S,Q)∼p(T ) [L(S,Q,Θ)] , L(S,Q,Θ) = 1 |Q| ∑
  (x,y)∼Q
  − log p(y|x, S; Θ) + λΩ(Θ), (1)
  where p(T ) is the distribution of tasks, Θ are the parameters of the model and p(y|x, S; Θ) is the probability assigned by the model to label y of query example x (given the support set S), and Ω(Θ) is an optional regularization term on the model parameters with factor λ.
  Conventional few-shot classification targets the setting of N -way-K-shot, where the number of classes and examples are fixed in each episode. Popular benchmarks following this approach include Omniglot (Lake et al., 2015) or benchmarks made of subsets of ImageNet, such as miniImageNet (Vinyals et al., 2016) and tieredImageNet (Ren et al., 2018). In such benchmarks, the tasks for training cover a set of classes that is disjoint from the classes in the test set of tasks. However, with the training and test sets tasks coming from a single dataset/domain, the distribution of tasks found in either sets is similar and lacks variability, which may be unrealistic in practice.
  It is in this context that Triantafillou et al. (2020) proposed Meta-Dataset, as a further step towards large-scale, multi-domain few shot classification. Meta-Dataset includes ten datasets (domains), with eight of them available for training. Additionally, each task sampled in the benchmark varies in the number of classes N , with each class also varying in the number of shots K. As in all few-shot learning benchmarks, the classes used for training and testing do not overlap.
sections.heading: 2.2 BACKGROUND AND RELATED WORK
sections.text: Meta-Learning A promising approach for few-shot classification is to use meta-learning to more directly train a model to learn to perform few-shot classification, in an end-to-end way. The two most popular methods are Prototypical Networks (Snell et al., 2017) and Model Agnostic Meta-Learning (MAML) (Finn et al., 2017). Triantafillou et al. (2020) showed that prototypical networks and MAML could be combined by leveraging prototypes for the initialization of the output weights value in the inner loop. Requeima et al. (2019) also proposed Conditional Neural Adaptive Processes (CNAPs) for few-shot classification, which can be seen as extending prototypical networks with a more sophisticated architecture that allows for improved task adaptation. This architecture was later improved further by Bateni et al. (2020) with Simple CNAPS, leading to one of the current best methods on Meta-Dataset. Another line of work which leverages the idea of “transfer by fine-tuning” can be found in Appendix B.
  Universal Representations In contrast, our work instead builds on that of Dvornik et al. (2020) and their method SUR (Selecting from Universal Representations). Bilen & Vedaldi (2017) introduced the term universal representation to refer to a representation that supports good performance in multiple domains. One proposal towards such a representation is to train different neural networks backbones separately on the data of each available domain, then simply to concatenate the representation learned by each. Another is to introduce some parameter sharing between the backbones, by having a single network conditioned on the domain of the provenance of each batch of training data (Rebuffi et al., 2018), e.g. using Feature-wise Linear Modulate (FiLM) (Perez et al., 2018). SUR proposes to leverage a universal representation in few-shot learning tasks with a feature selection procedure that assigns different weights to each of the domain-specific subvectors of the universal representation. The objective is to assign high weights only to the domain-specific representations that are specifically useful for each few-shot task at hand. The weights are inferred by optimizing a loss on the support set that encourages high accuracy of a nearest-centroid classifier. As such, the method does not involve any meta-learning—a choice motivated by the concern that meta-learning may struggle in generalizing to domains that are dissimilar to the training domains. SUR achieved some of the best performances on Meta-Dataset. However, a contribution of our work is to provide evidence that meta-learning can actually be used to replace SUR’s hand-designed inference procedure and improve performance further.
  Task Adaptive Representations Another line of work tries to retrieve task adaptive representations for each task. Task specific representations can be conditioned on a representation of the current task (Oreshkin et al., 2018; Wang et al., 2019), projected to another space (Yoon et al., 2019), or masked based on inter-class commonality and inter-class uniqueness (Li et al., 2019). While the representation extracted from URT is also task adaptive, it is adaptive to a set of pretrained backbones and can be applied to more complicated multi-domain scenarios. Wang & Hebert (2016) proposed to improve a CNN by adding extra layers and train it using unsupervised data while our contribution mainly lies in composing representations instead of an improved CNN. Alet et al. (2018) introduced a modular meta-learning method, which learns a repertoire of modules that serves as nodes to construct a tree structure to solve a new robotic-related task. Comparatively, URT is a one-for-all layer which doesn’t need to construct different module structures for each task.
  Transformer Networks Our meta-learning approach to leverage universal representations is inspired directly from Transformer networks (Vaswani et al., 2017). Our model structure is inspired by the structure of the dot-product self-attention in the Transformer, which we adapted here to multidomain few-shot learning by designing appropriate parametrizations for queries, keys and values. Self-attention was explored in the single-domain training regime by Ye et al. (2020); Liu et al. (2019b;a; 2020), however for a different purpose, where each representation of individual examples in a task support set is influenced by all other examples. Rather than using self-attention between individual examples in the support set, our model uses self-attention to select between different domain-specific backbones.
sections.heading: 3 UNIVERSAL REPRESENTATION TRANSFORMER LAYER
sections.text: In this section, we describe our proposed URT layer, which uses meta-learning episodic training to learn how to combine the domain-specific backbones of a universal representation for any given fewshot learning classification task. URT layer can be built on top of any set of pretrained backbones
  without further costly fine-tuning of the backbones. More details on how to train multiple domainspecific backbones can be found in Appendix C.
  Conceptually, the proposed model views the support set S of a task as providing information on how to query and retrieve from the set {ri} of m pre-trained backbones the most appropriate backbone to build an adapted representation φ for the task.
  We would like the model to support a variety of strategies on how to retrieve backbones. For example, it might be beneficial for the model to retrieve a single backbone from the set, especially if the domain of the given task matches perfectly that of a domain found in the training set. Alternatively, if some of the training domains benefit from much more training data than others, a better strategy might be to attempt some cross-domain generalization towards the few-shot learning task by blending many backbones together, even if none matches the domain of the task perfectly.
  This motivates us to use dot-product self-attention, inspired by layers of Transformer networks (Vaswani et al., 2017). For this reason, we refer to our model as a Universal Representation Transformer (URT) layer. Additionally, since each class of the support set might require a different strategy, we perform attention separately for each class and their support set Sc = {x|(x, y) ∈ S and y = c}.
sections.heading: 3.1 SINGLE-HEAD URT LAYER
sections.text: We start by describing an URT layer consisting of a single attention head. An illustration of a singlehead URT layer is shown in Figure 1. Let ri(x) be the output vector of the backbone for domain i. We then write the universal representation as
  r(x) = concat(r1(x), . . . , rm(x)). (2)
  This representation provides a natural starting point to obtain a representation of a support set class. Specifically, we will note
  r(Sc) = 1 |Sc| ∑ x∈Sc r(x) (3)
  as the representation for the set Sc. From this, we can describe the URT layer by defining the queries1, keys, the attention mechanism and output of the layer:
  Queries qc: For each class c, we obtain a query through qc = Wqr(Sc) + bq , where we have a learnable query linear transformation represented by matrix Wq and bias bq .
  Keys ki,c: For each domain i and class c, we define keys as ki,c = Wkri(Sc)+bk, using a learnable linear transformation Wk and bk and where ri(Sc) = 1/|Sc| ∑ x∈Sc ri(x), using a similar notation as for r(Sc). 1Unable to avoid the unfortunate double usage of the term ”query” due to conflicting conventions, we highlight the difference between the query sets Q of few-shot tasks and the queries qc of an attention mechanism.
  Algorithm 1 Training of URT layer Input: Number of tasks τtotal, m pre-trained backbones ;
  1: for τ ∈ {1, · · · , τtotal} do 2: Sample a few-shot task T with support set S and query set Q; 3: # Infer adapted representation for task from S 4: For each class, obtain representation using m pre-trained backbones as in Eq. (3); 5: Obtain attention scores using Eq. (4,5) for each head using support set S; 6: # Use adapted representation to predict labels in Q from support set S 7: Compute adapted representation of examples in S and Q as in Eq. (6,7); 8: Compute probabilities of label of examples in Q using Prototypical Network as in Eq. (9); 9: Compute loss as in Eq. (1,8) and perform gradient descent step on URT parameters Θ;
  10: end for
  Attention scores αi: as for regular Transformer layers, we use scaled dot-product attention
  αi,c = exp(βi,c)∑ i′ exp(βi′,c) , βi,c = qc >ki,c√ l , (4)
  where l is the dimensionality of the keys and queries. Then, these per-class scores are aggregated to obtain scores for the full support set by averaging
  αi = ∑ c αi,c N . (5)
  Equipped with these attention scores, the URT layer can now produce an adapted representation for the task (for the support and query set examples) by computing
  φ(x) = ∑ i αiri(x) . (6)
  As we can see, this approach has the flexibility of either selecting a single domain-specific backbone (by assigning αi = 1 for a single domain) or blending different domains together (by having αi >> 0 for multiple backbones).
sections.heading: 3.2 MULTI-HEAD URT LAYER
sections.text: The URT layer described so far can only learn to retrieve a single backbone (or blending of backbones). Yet, it might be beneficial to retrieve multiple different (blended) backbones, especially for a few-shot task that would include many classes of varying complexity.
  Thus, to achieve such diversity in the adapted representation, we also consider URT layers with multiple heads, i.e. where each head corresponds to the calculation of Equation 6 and each head has its own set of parameters (Wq,bq,Wk,bk). Denoting each head now as φh, a multi-head URT layer then produces as its output the concatenation of all of its heads:
  φ(x) = concat(φ1(x), . . . , φH(x)). (7)
  Empirically we found that the randomness in the initialization of head weights alone did not lead to uniqueness and being complimentary between the heads, so inspired by Lin et al. (2017), we add a regularizer to avoid duplication of the attention scores:
  Ω(Θ) = ‖(AA> − I)‖F 2 , (8)
  where ‖ · ‖F is the Frobenius norm of a matrix and A ∈ Rn×m is the matrix for attention scores, with Ah being the vector of all scores αi for head h. The identity matrix I regularizes each set of attention scores to be more focused so that multiple heads can attend to different domain-specific backbones.
sections.heading: 3.3 TRAINING STRATEGY
sections.text: We train representations produced by the URT layer by following the approach of Prototypical Networks (Snell et al., 2017), where the probability of a label y for a query example x given the
  support set of a task is modeled as:
  p(y = c|x, S; Θ) = exp(−d(φ(x)− pc))∑N c′=1 exp(−d(φ(x)− pc′)) , (9)
  where d is a distance metric and pc = 1/|Sc| ∑
  x∈Sc φ(x) corresponds to the centroid of class c, referred to as its prototype. We use (negative) cosine similarity as the distance. The full training algorithm is presented in Algorithm 1.
sections.heading: 4 EXPERIMENTS
sections.text: In this section, we seek to answer three key experimental questions:
  Q1 How does URT compare with previous state-of-the-art on Meta-Dataset for multi-domain fewshot classification?
  Q2 Do the URT attention heads generate interpretable and meaningful attention scores? Q3 Does the URT layer provide consistent benefits, even when pre-trained backbones are trained in
  different ways?
  In addition, we investigate architectural choices made, such as our models for keys/queries and their regularization, and study their contribution to achieving strong performance with URT.
sections.heading: 4.1 DATASETS AND SETUP
sections.text: We test our methods on the large-scale few-shot learning benchmark Meta-Dataset (Triantafillou et al., 2020). It consists of ten datasets with various data distributions across different domains, including natural images (Birds, Fungi, VGG Flower), hand-written characters (Omniglot, Quick Draw), and human created objects (Traffic Signs, Aircraft). Among the ten datasets, eight provide data that can be used during either training, validation and testing (with each class assigned to only one of those sets), while two datasets are solely used for testing. Following Bateni et al. (2020); Requeima et al. (2019), we also report results on MNIST (LeCun et al., 1998), CIFAR10 and CIFAR100 (Krizhevsky et al., 2009) as additional unseen test datasets. Following Triantafillou et al. (2020), few-shot tasks are sampled with varying number of classesN , varying number of shots K and class imbalance. The performance is reported as the average accuracy over 600 sampled tasks. More details of Meta-Dataset can be found in Triantafillou et al. (2020).
  The domain-specific backbones are pre-trained following the setup in (Dvornik et al., 2020). Then, we freeze the backbone and train the URT layer for 10,000 episodes, with an initial learning rate of 0.01 and a cosine learning rate scheduler. Following Chen et al. (2020), the training episodes have 50% probability coming from the ImageNet data source. Since different pre-trained backbones may produce representations with different vector norms, we normalize the outputs of the backbones as in Dvornik et al. (2020). URT is trained with parameter weight decay of 1e-5 and with a regularization factor λ = 0.1. The number of heads (H in Equation 7), is set to 2 and the dimension of the keys and
  queries (l in Equation 4) is set to 1024. We choose the hyper-parameters based on the performance of the validation set. Details of the hyper-parameter selection and how the performance is influenced by them are outlined in Section 4.5. We find that the bottleneck of training URT is extracting features from CNN. Since we freeze the CNN when training the URT, we find dumping the extracted feature episodes can significantly speed up the training procedure from days to around 2 hours.
sections.heading: 4.2 COMPARISON WITH PREVIOUS APPROACHES
sections.text: Table 1 presents a comparison of URT with SUR, as well as other baselines based on transfer learning by fine-tuning (Saikia et al., 2020) or meta-learning (Prototypical Networks (Snell et al., 2017), first-order MAML (Finn et al., 2017), ProtoMAML (Triantafillou et al., 2020), CNAPs (Requeima et al., 2019)) and Simple CNAPS(Bateni et al., 2020).
  We observe in Table 1 that URT establishes a new state-of-the-art on Meta-Dataset, by achieving the top performance on 8 out of the 10 dataset sources. When comparing to its predecessor, URT outperforms SUR on 4 datasets without compromising performance on others, which is challenging to achieve in the multi-domain setting. Of note, the average inference time for URT is 0.04 second per task, compared to 0.43 for SUR, on a single V100. Thus, getting rid of the optimization procedure for every episode with our meta-trained URT layer also significantly increases the latency, by more than 10×. More results on additional datasets can be found in Appendix A.
sections.heading: 4.3 INTERPRETING AND VISUALIZING ATTENTION BY URT
sections.text: To better understand how the URT model of Section 4.2 uses its two heads to build adapted representations, we visualize the attention scores produced on the test tasks of Meta-Dataset in Figure 2.
  The blue (first head) and orange (second head) heatmaps summarize the values of the attention scores (Equation 5), averaged across several tasks for each test domain. Specifically, the element on row t and column i is the averaged attention scores αi computed on test set domain t for the backbone from domain i. Note that the last two rows are the two unseen domain datasets. We found that for datasets from the seen domains, i.e. the first eight rows, one head (right, orange) consistently puts most of its weight on the backbone pre-trained on the same domain, while the other head (left, blue) learns relatively smoother weight distributions that blend other related domains. For unseen datasets, the right head puts half of its weight on ImageNet and the left head learned to blend the representations from four backbones.
sections.heading: 4.4 URT USING FILM MODULATED BACKBONES
sections.text: As additional evidence of the benefit of URT on universal representations, we also present experiments based on a different set of backbone architectures. Following SUR (Dvornik et al., 2020), we consider the backbones from a parametric network family, obtained by training a base backbone on one dataset (ILSVRC) and then learning separate FiLM layers (Perez et al., 2018) for each other dataset, to modulate the backbone so it is adapted to the other domains. These backbones collectively have only 0.5% more parameters than a single backbone. More details of the backbones can be found in Appendix C.
  A comparison between SUR and URT using these backbones (referred to as SUR-pf and URT-pf) is presented in Table 2. Once again, URT can improve the performance on VGG Flower without sacrificing performance on others.
sections.heading: 4.5 HYPER-PARAMETER AND ABLATION STUDIES
sections.text: We analyze the importance of the various components of URT’s attention mechanism structure and training strategy in Table 3. First we analyze the importance of using the support set to model queries and/or keys. To this end, we consider setting the matrices Wq / Wk of the query / key linear transformation to 0, which only leaves the bias term. We found that the support set representation is most crucial for building the keys (row w/o Wk in the table) and has minor benefits for queries (row w/o Wq) in the table. This observation is possibly related to the success of attention-based models with learnable constant queries (Liu et al., 2016; Lin et al., 2017). We also found that adding a regularizer Ω(Θ) as in Equation 8 is important for some datasets, specifically VGG Flower and Birds.
  An important hyper-parameter in URT is the number of heads H . We chose this hyper-parameter based on the performance on validation set of tasks in Meta-Dataset. In Table 4, we show the validation performance of URT for varying number of heads. As suggested by Triantafillou et al. (2020), we considered looking at the rank of the performance achieved by each choice of H for each validation domains, and taking the average across domains as a validation metric. However,
  since the performances when using two to four heads are similar and yield the same average rank, we instead simply consider the average accuracy as the selection criteria.
  In general, we observe a large jump in performance when using multiple heads instead of just one. However, since the number of heads controls the capacity, predictably we also observe that having too many heads leads to overfitting.
sections.heading: 5 CONCLUSION
sections.text: We proposed the URT layer to effectively integrate representations from multiple domains and demonstrated improved performance in multi-domain few-shot classification. Notably, our URT approach was able to set a new state-of-the-art on Meta-Dataset, and never performs worse than its predecessor (SUR) while also being 10× more efficient at inference. This work suggests that combining meta-learning with pre-trained universal representations is a promising direction for new few-shot learning methods. Specifically, we hope that future work can investigate the design of richer forms of universal representations that go beyond simply pre-training a single backbone for each domain, and developing meta-learners adapted to those settings.
sections.heading: A EXPERIMENTS ON MORE DATASETS
sections.text: We also report performances on the MNIST, CIFAR-10 and CIFAR-100 dataset sources in Table 5, and compare with the subset of methods that have reported on these datasets. There, URT neither improves nor gets worse performance than SUR, yeilding top performance on the MNIST domain but not on the CIFAR-10/CIFAR-100 domain, on which Simple CNAPS has the best performance.
sections.heading: B MORE RELATED WORKS
sections.text: Transfer by fine-tuning A simple and effective method for few-shot classification is to perform transfer learning by first learning a neural network classifier on all data available for training and using its representation to initialize and then fine-tune neural networks on the few-shot classification tasks found at test time (Chen et al., 2019; Triantafillou et al., 2020; Dhillon et al., 2020; Saikia et al., 2020). Specifically, Saikia et al. (2020) have shown that competitive performance can be reached using a strong hyper-parameter optimization method applied on a carefully designed validation metric appropriate for few-shot learning.
sections.heading: C STRUCTURES AND TRAINING STRATEGY OF BACKBONES
sections.text: Our URT layer can be built on top of a set of pretrained backbones. The structures of the backbones follow the approach in Dvornik et al. (2020). For Table 1, we use the ResNet18 architecture (He et al., 2016) for the backbones, where a separate backbone is pretrained on each domain separately using the corresponding training data in meta-dataset. The training domains include ImageNet, Omniglot, Aircraft, CU-Birds, Textures, Quick Draw, Fungi and VGG-Flower. For the parametric network family in Table 2, a base ResNet18 is first trained on ImageNet. Then a small number of modulating parameters are trained on the other domains, using their domain-specific training data, while the rest of the base backbone’s weights stay fixed. Specifically, FiLM feature modulation (Perez et al., 2018) is used. This type of parametric network family thus allows for a much reduced number of learnable parameters, by reusing the weights from the base network. In experiments, we use the pretrained backbones released by Dvornik et al. (2020) for both cases, without
  any further finetuning. End-to-end training of a set of backbones and the URT layers requires unaffordable computational cost, so we fix the pretrained backbones and only train the URT layer.
  Implementation details for training backbones The training details of the backbones come from Dvornik et al. (2020). For optimization, SGD with momentum was used, using cosine learning rate annealing. Since the datasets come from different domains, the starting learning rate, the maximum number of training iterations and annealing frequency are set individually for each dataset. Data augmentation is applied and a constant weight decay of 7×10−4 is set. For each dataset, a grid search over batch size in [8, 16, 32, 64] was run and the one that maximizes accuracy on the validation set was picked. For the parametric network family, the base ResNet18 trained on ImageNet is the same. For other backbones, cosine annealing as learning rate policy is also used, weight decay and data augmentation employed as above. Please refer to Table 4 and Table 5 in the original SUR paper (Dvornik et al., 2020) for the specific values of the hyperparameters for the individual feature networks and the parametric network family, respectively.
sections.heading: D RESULTS ON TRAFFIC SIGNS
sections.text: The shuffle buffer bug described in meta-dataset issue #54 (https://github.com/googleresearch/meta-dataset/issues/54) has been propagated to previous works such as CNAPs (Requeima et al., 2019). The results for the Traffic Signs dataset are considerably worse after fixing this bug. For instance, URT degrades from 69.4±0.8 to 51.1±1.1. Please visit the official meta-dataset GitHub repo for more details. The main paper shows the corrected results (for URT and competing approaches). For completeness, we also provide the tables as they were in the initial version of this paper for your reference in this Appendix. Both sets of results overall support the advantageous performance of URT over previous work.
title: A UNIVERSAL REPRESENTATION TRANSFORMER LAYER FOR FEW-SHOT IMAGE CLASSIFICATION