summaries: The paper proposes to apply the task-aware-modulation on global Meta-learning for learning on tasks sampled from heterogeneous distributions. This paper builds on prior work on the feature-based task characterization to integrate the rehearsed task gradient descent trajectory into task representation. As the rehearsal task is computationally expensive, the authors learn a different network to estimate the rehearsed task-trajectory characterization from the feature representation. The proposed framework is tested on few-shot image classification (meta dataset and miniimagenet) and cold-start recommendation tasks.

summaries: This paper proposes a meta-learning approach based on MAML but with a task-conditioned initialization, which takes into account both information extracted from the features of the task (support set) data, as well as geometric information from a “rehearsed” learning path (obtained by gradient descent on the support set). The geometric information includes the parameters, loss, gradient and fisher matrix for different steps of the task adaptation process. They propose a GRU architecture to process this information across the different steps, yielding a path embedding. For both the feature and path embeddings, they utilize clustering, motivated by the need to share knowledge across similar tasks. They then combine the path and feature embeddings via an additional neural network to generate the final task-specific initialization. Computing the path embedding requires rehearsing which is computationally expensive (it’s akin to training twice on the task, once for rehearsing, and then training from the task-conditioned initialization). To amend this at inference time, they meta-learn a ‘tunnel’ connection that predicts the path embedding from the feature embedding. At test time, only a forward pass through this meta-learned connection is required instead of rehearsing. They experimentally evaluate their approach on few-shot classification and cold-start recommender problems and show performance gains over similar MAML-based approaches. They also perform ablation studies and analyses to understand the contribution of different parts of this system to downstream performance.

summaries: This paper proposes a clustered task-aware meta-learning algorithm. The proposed algorithm firstly collects the learning path and uses the path to train a sequence module. The sequence module output is combined with the task feature. The task feature is derived from the weighted sum of the soft cluster centers. The combination is then used as initialization of the model parameter for task adaptation. Experiments on image classification and cold-start recommendation demonstrate superior performance compared to baseline algorithms.

summaries: This paper looks at the problem of meta-learning with heterogeneous tasks. This is an interesting problem that is receiving increasing attention in the past two or three years. Several previous work address this problem by clustering task representations and let the meta-learner exploit the information about task cluster. The authors claim that, for the first time, they cluster tasks not only at the level of the input representation (features), but also at the level of the optimization trajectory in parameter space.

abstractText: To enable effective learning of new tasks with only few samples, meta-learning acquires common knowledge from the existing tasks with a globally shared metalearner. To further address the problem of task heterogeneity, recent developments balance between customization and generalization by incorporating task clustering to generate the task-aware modulation to be applied on the global meta-learner. However, these methods learn task representation mostly from the features of input data, while the task-specific optimization process with respect to the baselearner model is often neglected. In this work, we propose a Clustered TaskAware Meta-Learning (CTML) framework with task representation learned from both features and learning path. We first conduct rehearsed task learning from the common initialization, and collect a set of geometric quantities that adequately describes this learning path. By inputting this set of values into a meta path learner, we automatically abstract path representation optimized for the downstream clustering and modulation. To further save the computational cost incurred by the additional rehearsed learning, we devise a shortcut tunnel to directly map between the path and feature cluster assignments. Extensive experiments on two real-world application domains: few-shot image classification and cold-start recommendation demonstrate the superiority of CTML compared to state-of-the-art baselines.

authors:
Alessandro Achille
Michael Lam
Rahul Tewari
Avinash Ravichandran
Subhransu Maji
Charless C Fowlkes
Stefano Soatto
Pietro Perona

title: Task2vec: Task embedding for meta-learning

authors:
Rahaf Aljundi
Francesca Babiloni
Mohamed Elhoseiny
Marcus Rohrbach
Tinne Tuytelaars

title: Memory aware synapses: Learning what (not) to forget

authors:
Sungyong Baik
Seokil Hong
Kyoung Mu Lee

title: Learning to forget for meta-learning

authors:
Peyman Bateni
Raghav Goyal
Vaden Masrani
Frank Wood
Leonid Sigal

title: Improved fewshot visual classification

authors:
Wei-Yu Chen
Yen-Cheng Liu
Zsolt Kira
Yu-Chiang Frank Wang
Jia-Bin Huang

title: A closer look at few-shot classification

venue: In International Conference on Learning Representations,

authors:
Mircea Cimpoi
Subhransu Maji
Iasonas Kokkinos
Sammy Mohamed
Andrea Vedaldi

title: Describing textures in the wild

authors:
Arnout Devos
Yatin Dandi

title: Model-agnostic learning to meta-learn

venue: arXiv preprint arXiv:2012.02684,

authors:
Manqing Dong
Feng Yuan
Lina Yao
Xiwei Xu
Liming Zhu

authors:
Chelsea Finn
Pieter Abbeel
Sergey Levine

venue: In International Conference on Machine Learning,

authors:
Chelsea Finn
Kelvin Xu
Sergey Levine

title: Probabilistic model-agnostic meta-learning

authors:
Sebastian Flennerhag
Pablo G Moreno
Neil D Lawrence
Andreas Damianou

title: Transferring knowledge across learning processes

authors:
Jezabel R Garcia
Federica Freddi
Feng-Ting Liao
Jamie McGowan
Tim Nieradzik
Da-shan Shiu
Ye Tian
Alberto Bernacchia

title: Meta-learning with maml on trees

venue: arXiv preprint arXiv:2103.04691,

authors:
Francesco Gianfelici
Viviana Battistelli

venue: IEEE Transactions on Information Theory,

authors:
F Maxwell Harper
Joseph A Konstan

title: The movielens datasets: History and context

authors:
Sameeksha Katoch
Kowshik Thopalli
Jayaraman J Thiagarajan
Pavan Turaga
Andreas Spanias

authors:
James Kirkpatrick
Razvan Pascanu
Neil Rabinowitz
Joel Veness
Guillaume Desjardins
Andrei A Rusu
Kieran Milan
John Quan
Tiago Ramalho
Agnieszka Grabska-Barwinska

title: Overcoming catastrophic forgetting in neural networks

venue: Proceedings of the national academy of sciences,

authors:
Hoyeop Lee
Jinbae Im
Seongwon Jang
Hyunsouk Cho
Sehee Chung

authors:
Huaiyu Li
Weiming Dong
Xing Mei
Chongyang Ma
Feiyue Huang
Bao-Gang Hu

venue: In International conference on machine learning,

authors:
Xixun Lin
Jia Wu
Chuan Zhou
Shirui Pan
Yanan Cao
Bin Wang

venue: arXiv preprint arXiv:2103.06137,

authors:
Yuanfu Lu
Yuan Fang
Chuan Shi

authors:
Subhransu Maji
Esa Rahtu
Juho Kannala
Matthew Blaschko
Andrea Vedaldi

title: Fine-grained visual classification of aircraft

venue: arXiv preprint arXiv:1306.5151,

authors:
Julian McAuley
Christopher Targett
Qinfeng Shi
Anton Van Den Hengel

title: Image-based recommendations on styles and substitutes

authors:
Maria-Elena Nilsback
Andrew Zisserman

authors:
Boris N Oreshkin
Pau Rodrı́guez López
Alexandre Lacoste

authors:
James Requeima
Jonathan Gordon
John Bronskill
Sebastian Nowozin
Richard E Turner

venue: Advances in Neural Information Processing Systems

authors:
Matthew Riemer
Ignacio Cases
Robert Ajemian
Miao Liu
Irina Rish
Yuhai Tu
Gerald Tesauro

authors:
Andrei A Rusu
Dushyant Rao
Jakub Sygnowski
Oriol Vinyals
Razvan Pascanu
Simon Osindero
Raia Hadsell

title: Meta-learning with latent embedding optimization

authors:
B Schroeder
Y Cui

title: Fgvcx fungi classification challenge

authors:
Jake Snell
Kevin Swersky
Richard Zemel

title: Prototypical networks for few-shot learning

venue: In Advances in Neural Information Processing Systems,

authors:
Johannes Stallkamp
Marc Schlipsing
Jan Salmen
Christian Igel

venue: Neural networks,

authors:
Qianru Sun
Yaoyao Liu
Tat-Seng Chua
Bernt Schiele

title: Meta-transfer learning for few-shot learning

authors:
Qiuling Suo
Jingyuan Chou
Weida Zhong
Aidong Zhang

authors:
Sebastian Thrun
Lorien Pratt

title: Learning to learn: Introduction and overview

venue: In Learning to learn,

authors:
Yonglong Tian
Yue Wang
Dilip Krishnan
Joshua B Tenenbaum
Phillip Isola

venue: In Computer Vision–ECCV 2020: 16th European Conference,

authors:
Manasi Vartak
Arvind Thiagarajan
Conrado Miranda
Jeshua Bratman
Hugo Larochelle

authors:
Ashish Vaswani
Noam Shazeer
Niki Parmar
Jakob Uszkoreit
Llion Jones
Aidan N Gomez
Łukasz Kaiser
Illia Polosukhin

title: Attention is all you need

venue: In Advances in neural information processing systems,

authors:
Oriol Vinyals
Charles Blundell
Timothy Lillicrap
Koray Kavukcuoglu
Daan Wierstra

title: Matching networks for one shot learning

authors:
Maksims Volkovs
Guangwei Yu
Tomi Poutanen

title: Dropoutnet: addressing cold start in recommender systems

authors:
Risto Vuorio
Shao-Hua Sun
Hexiang Hu
Joseph J. Lim

venue: In Neural Information Processing Systems,

authors:
Junyuan Xie
Ross Girshick
Ali Farhadi

title: Unsupervised deep embedding for clustering analysis

authors:
Huaxiu Yao
Ying Wei
Junzhou Huang
Zhenhui Li

title: Hierarchically structured meta-learning

authors:
Huaxiu Yao
Xian Wu
Zhiqiang Tao
Yaliang Li
Bolin Ding
Ruirui Li
Zhenhui Li

title: Automated relational meta-learning

authors:
Jaesik Yoon
Taesup Kim
Ousmane Dia
Sungwoong Kim
Yoshua Bengio
Sungjin Ahn

title: Bayesian model-agnostic meta-learning

authors:
Runsheng Yu
Yu Gong
Xu He
Yu Zhu
Qingwen Liu
Wenwu Ou
Bo An

authors:
Tianhe Yu
Saurabh Kumar
Abhishek Gupta
Sergey Levine
Karol Hausman
Chelsea Finn

title: Gradient surgery for multi-task learning

venue: arXiv preprint arXiv:2001.06782,

authors:
Friedemann Zenke
Ben Poole
Surya Ganguli

title: Continual learning through synaptic intelligence

text:
1 INTRODUCTION The astonishing performance of deep learning relies on large amounts of data, which are not always available. Humans, on the other hand, are able to learn new tasks much more quickly, leveraging prior experience to relate knowledge among tasks. Inspired by this property of human intelligence, meta-learning (also known as learning to learn) (Thrun & Pratt, 1998) acquires transferable commonalities from existing tasks for more efficient learning of new tasks. Although recent advances in meta-learning have demonstrated success in fields like few-shot image classification and cold-start recommendation (Vinyals et al., 2016; Snell et al., 2017; Ravi & Larochelle, 2016; Munkhdalai & Yu, 2017), most of them typically assume that all the tasks are drawn from a single distribution and face the challenge of handling tasks that come from different underlying distributions, a problem known as task heterogeneity (Devos & Dandi, 2020; Yao et al., 2020; Suo et al., 2020).
To overcome this challenge, many recently developed works leverage task-specific information to customize the global meta-learner (Vuorio et al., 2019; Oreshkin et al., 2018; Requeima et al., 2019; Finn et al., 2018; Yoon et al., 2018; Li et al., 2019; Rusu et al., 2018). To further consider generalization among related tasks, methods that perform various types of task clustering are proposed (Yao et al., 2019; 2020; Suo et al., 2020; Lu et al., 2020; Dong et al., 2020; Lin et al., 2021). Despite their effectiveness in improving over the globally shared meta-learning algorithms, they learn task representations only based on the input data distribution in (original or projected) feature space, while the interaction between data and the base-learner is often neglected.
The amount of information contained in the task-specific data about a network that is responsible for performing the task can be seen as a good representation of the task itself. This can be manifested as the gradients of the network parameters with respect to the task-specific loss, or Fisher Information Matrix (FIM) which indicates the importance of different network parameters in solving the task. Achille et al. (2019) introduce a task embedding method based on FIM to assist the meta task of selecting a pre-trained feature extractor. To alleviate the conflict issue of the global initialization methods, Baik et al. (2020) utilize task gradients at the initialization as task representation to produce
attenuation. However, these methods represent task based on gradients at only a single point in parameter space (e.g., at the initialization), while the potential of exploiting a wider range on the task manifold remains unexplored .
For the parameter initialization approach, the key of success lies in that the task adaptation process is accounted for when training the meta-learner. This task-specific learning may involve multiple gradient descent steps, and thus constitute a learning trajectory on the loss surface (Flennerhag et al., 2018; Zenke et al., 2017). To better characterize the task optimization behaviors, it is more beneficial to look at the complete learning trajectory as opposed to only the first gradient step at the initialization, as it is likely that tasks with similar gradients at first will have their learning paths diverged as the update proceeds. With that in mind, we are motivated to leverage the entire learning path for better task representation, which will then be used to condition the global initialization.
In particular, we propose a Clustered Task-aware Meta-Learning (CTML) framework, building upon the well-known global initialization method Model-Agnostic Meta-Learning (MAML) (Finn et al., 2017). To address the problem of task heterogeneity with a good balance between customization and generalization, we modulate the common initialization based on task representation learned from both its local task-specific information and global clustering results. In addition to using the input features to represent task, we further leverage the task learning path with respect to the baselearner to characterize task from the perspective of optimization. To facilitate clustering, we devise a GRU-based meta path learner which automatically abstracts path representations from the step-wise geometric quantities. We realize that it may be too costly to rehearse the entire learning process for task representation. Hence, we further propose a shortcut tunnel to bypass the rehearsed task learning during meta-testing and predict path cluster assignment directly from the feature cluster assignment. We carefully study the effectiveness of CTML in two real-world application domains: few-shot image classification and cold-start recommendation, and show that our method is able to outperform the baselines with comparable inference time.
2 RELATED WORK
To address the problem of task heterogeneity, recent developments often tailor the shared metalearner to different tasks using task-specific information. Finn et al. (2018) and Yoon et al. (2018) model the uncertainty exists in task distribution with probabilistic framework. Oreshkin et al. (2018), Li et al. (2019), Requeima et al. (2019) and Bateni et al. (2020) condition the base network on taskspecific data by designing a meta adaptation network. To enable more robust training, Rusu et al. (2018) propose to learn a lower-dimensional latent space specific to each task to generate the base network parameters. Relevant to our work, Vuorio et al. (2019) build upon MAML and modulate the global initialization based on the task representation learned from an encoder network.
However, customizing the common knowledge to individual task without considering the relations among similar tasks may lead to poor generalization. In regard to this, Dong et al. (2020) and Lin et al. (2021) apply K-means clustering on users (treated as tasks) based on their profile information to address the cold-start problem in recommender systems. Yao et al. (2019) employ a hierarchical structure to model the nested relationships inherent in domains with clear taxonomy, such as image classification. Yao et al. (2020) further develop an automatic relational graph method by constructing a meta-knowledge graph to preserve and propagate the global structural information. Suo et al. (2020) take advantages of an external knowledge base to facilitate task clustering. Despite their effectiveness in generalizing across tasks, they rely solely on input features for task representation, while the interaction between task data and the base-learner (e.g., gradients) is often neglected.
The idea of using gradients for task representation is commonly adopted in continual learning (Kirkpatrick et al., 2017; Zenke et al., 2017; Aljundi et al., 2018). In the context of task-aware metalearning, Achille et al. (2019) propose to use the FIM of a “probe” network to represent task, and Baik et al. (2020) leverage gradients of the globally initialized parameters to generate task-specific attenuation. However, these methods utilize gradients only at a single point in parameter space, whereas meta-learning often incorporates the entire task learning trajectory while learning the metalearner. Garcia et al. (2021) perform task clustering at each adaptation step and aggregate the gradients for more stable task adaptation. Flennerhag et al. (2018) highlight the benefits of leveraging the learning paths for deriving the common knowledge, but the proposed algorithm only relies on one geometric quantity – the length of the path, which can be limited in characterizing the learning paths, and the initialization is not tailored to specific task. Our work serves to address the above-mentioned
limitations, whereby higher-order behaviours of the learning path are also taken into account to learn a better task representation for modulating the global initialization.
3 PRELIMINARIES AND PROBLEM STATEMENT In task-heterogeneous setting, tasks {T1, T2, ..., TN} are sampled from a mixture of task distributions {p1(T ), p2(T ), ...}, where the number of underlying distributions is usually unknown. The goal of meta-learning is to learn the sharable knowledge by training the meta-learner on a set of metatraining tasks Ttr = {Ti}N tr
i=1 , and test it on a set of meta-testing tasks Tte = {Ti}Ni=Ntr+1. For each task Ti ∈ Ttr ∪Tte, its samples are further divided into a training set (also termed support set) DtrTi = {(xi,j , yi,j)} ntrTi j=1 and a test set (also termed query set) DteTi = {(xi,j , yi,j)}
nTi j=ntrTi +1 , which
together form an “episode” (Vinyals et al., 2016). This episodic scheme allows us to train tasks to learn fast during meta-training, and test the learning performance of new tasks in the same way during meta-testing. For few-shot learning, the size of the training set ntrTi is usually small.
Our work builds upon Model-Agnostic Meta-Learning (MAML) (Finn et al., 2017). It implements the meta-learner as an initialization of parameters θ0 ∈ RD of the base-learner fθ responsible for the prediction task. During meta-training, the global initialization θ0 is first adapted to each metatraining task Ti ∈ Ttr by learning on the respective training set DtrTi , which yields the task-specific parameters θTi . After that, loss is computed on the test set DteTi based on θTi and backward propagated to update θ0. Taking one step adaptation as an example, the meta-optimization is as follows:
θ∗0 = argmin θ0
∑
Ti∈Ttr L(fθTi ,D te Ti) = argmin θ0
∑
Ti∈Ttr L(fθ0−α▽θL(fθ0 ,DtrTi ),D te Ti), (1)
where α is the adaptation rate, L(fθ,D) can be mean square error loss for regression problem (i.e., 1|D| ∑ (x,y)∈D(y − fθ(x))2), or cross-entropy loss for classification problem (i.e., − 1|D| ∑ (x,y)∈D y log fθ(x)). During meta-testing, the learned initialization θ ∗ 0 is adapted to each meta-testing task Ti ∈ Tte using DtrTi , and the learning performance is evaluated on D te Ti .
However, with a globally shared initialization, MAML is not capable of handling task heterogeneity. Though task-specic methods have been developed to tailor the global initialization, they lack explicit modeling of the global clustering structure. Hence, a framework considering both task-specific information and global structure is desired.
4 METHODOLOGY Grounded on MAML, our CTML framework modulates the common initialization based on task representation learned from two different perspectives enhanced with the clustering information. Specifically, given the training set of a specific task, we first conduct rehearsed learning of the task from the common initialization and compute a set of step-wise quantities along the learning path. This set of values will be inputted into the meta path learner to generate a task-specific path embedding. On the other hand, input features of the given task will also be extracted to form the feature embedding. Both feature and path embeddings will then undergo soft K-means clustering in their respective latent space. Finally, the two embeddings enriched with cluster information will be aggregated to produce the task-aware modulation to be applied on the global initialization. From this modulated intialization, standard MAML follows, which conducts the actual task learning on the training set and perform meta-update across tasks using the test sets. To further improve the inference efficiency during meta-testing, a shortcut tunnel is used to bypass the rehearsed learning process and generate the path cluster assignment directly from the feature cluster assignment. An overview of CTML framework is shown in Figure 1.
In the following sections, we first elaborate on task representation learning based on learning path and features respectively, and then introduce the task-aware modulation and the shortcut tunnel.
4.1 TASK REPRESENTATION BASED ON LEARNING PATH
4.1.1 PATH CONSTRUCTION To obtain representation of task Ti based on learning path, we first conduct a τ -step rehearsed learning from the global initialization θ0 on training set DtrTi . Applying the same gradient descent update as in MAML, we obtain the updated parameters at each step t ∈ {1, 2, ..., τ} by:
θ̃tTi = θ̃ t−1 Ti − α▽θL(fθ̃t−1Ti ,D tr Ti), where θ̃ 0 Ti = θ0. The overhead ·̃ is used to differentiate between the parameters updated from the rehearsed learning and from the actual learning.
Joining the discrete points {(θ̃tTi , L̃ t Ti)} τ t=0 constitutes the rehearsed learning path of Ti (we write L(fθ̃tTi ,DtrTi) as L̃ t Ti for brevity) on the unique task manifold MTi ∈ R
D+1, characterized by the base-learner function and the task-specific data distribution. We collect the coordinates of point (θ̃tTi , L̃ t Ti) at each update step t ∈ {0, 1, 2, ..., τ} to indicate the exact learning trajectory. To account for higher-order behaviours of the learning path, we further incorporate the gradients ▽θL̃tTi and the Fisher Information Matrix (FIM) F̃ tTi at each step, which specify the direction and curvature respectively. Note that by definition, the FIM is equivalent to the Hessian matrix in expectation for cross-entropy loss: F̃ tTi = Ex,y∼Ti [▽θL(fθ̃tTi ,x, y)▽θL(fθ̃tTi ,x, y)
⊤] = Ex,y∼Ti [▽2θL(fθ̃tTi ,x, y)]. It provides a measure of the amount of information that task Ti contains about the parameters θ (Gianfelici & Battistelli, 2009). Following Achille et al. (2019), we assume negligible correlation among different parameters and consider only the diagonal entries of F̃ tTi , denoted by F̃ t Ti . As a result, we obtain a set of step-wise quantities {(θ̃tTi , L̃ t Ti ,▽θL̃ t Ti , F̃ t Ti)} τ t=0 that adequately describes the rehearsed learning path.
As opposed to looking at only a single point in the parameter space (e.g., use gradients at the initialization to represent task (Baik et al., 2020)), it is benificial to take into account the entire learning path traversed, which gives a more complete picture of the optimization process. Take for instance, two tasks Ta and Tb may have gradients forming a small angle ψ0a,b at the initialization, i.e., cos(ψ0a,b) > 0. However, it is likely that their learning paths will diverge as the gradient update proceeds, i.e., the accumulated angle ∑τ t=0 cos(ψ t a,b) < 0 (see dotted paths of T2 and T3 in Figure 2). Conversely, the tasks may have gradients pointing at different directions at first, but eventually converge towards the same direction as the learning proceeds (see dotted paths of T1 and T2). Considering a single step only can be restrictive for task representation from the optimization perspective. Looking further down the paths allows better characterization of the learning behaviors and even the flexibility to determine the “important” steps (as shown in experiments later), producing more informative task representations. Furthermore, leveraging path representations serves to provide a stronger inductive bias to condition the initialization in favorable direction, as the rehearsed learning path can be considered as encoding the quality of the initialization with respect to the task (i.e., tasks with longer and more zigzag paths imply that the initialization is not a good one for them).
4.1.2 PATH REPRESENTATION Having collected the step-wise quantities {(θ̃tTi , L̃ t Ti ,▽θL̃ t Ti , F̃ t Ti)} τ t=0 along the rehearsed learning path, the problem now is how to evaluate “similarity” among different task learning paths for clustering. Previous works with single-step gradients usually compute dot product or cosine similarity between gradients of two tasks (Riemer et al., 2018; Katoch et al., 2019; Yu et al., 2020; Baik et al., 2020). With multiple steps, a straightforward adaptation will be to simply sum up the similarity scores at all steps. However, this human-defined rule may not be the best way of measuring path similarity. Instead, we propose to employ a meta path learner to automatically learn a
path embedding from the step-wise quantities, and then measure similarity based on these vector representations. In other words, the meta path learner induces a latent space on which the distance metric best characterizes what is considered as “similar” (or “disimilar”) among task learning paths.
Before we delve into the design of the path learner, we first elaborate how we construct the functional input from the step-wise quantities. Recall that at each step t, we have θ̃tTi ,▽θL̃ t Ti , F̃ t Ti ∈ R
D and L̃tTi ∈ R. To obtain a regularly shaped input, we duplicate L̃ t Ti to form a D-dimensional vector and stack it together with the other 3 components to form a matrix PtTi ∈ R 4×D. Further stacking the τ + 1 steps gives the overall 3-D matrix PTi ∈ R(τ+1)×4×D. To avoid high model complexity, we apply the meta path learner coordinate-wise on the base-learner parameters. That is, the same path learner is used to process the matrix PdTi ∈ R
(τ+1)×4 independently for all d ∈ {1, 2, ..., D}. For the path learner design, we propose to leverage the Gated Recurrent Units (GRUs) to model the sequential dependencies among steps1. Specifically, the hidden state hd,ti at step t is obtained by inputting the t-th sliced vector Pdt,: ∈ R4 of PdTi and the hidden state h d,t−1 i at step t − 1 into the GRU cell: rd,ti = σ(Wr · [h d,t−1 i ,P d t,:]), z d,t i = σ(Wz · [h d,t−1 i ,P d t,:]),
h̃d,ti = tanh(Wh̃ · [r d,t i ⊙ h d,t−1 i ,P d t,:]), h d,t i = (1− z d,t i )⊙ h d,t−1 i + z d,t i ⊙ h̃ d,t i ,
(2)
where [·, ·] denotes concatenation, σ(·) is the sigmoid activation, rd,ti , z d,t i ∈ (0, 1) are gates that control how much of past and present information to be retained, and Wr,Wz,Wh̃ are learnable weights shared across all steps. For step t = 0, we use a zero-initialized input hidden state.
The path representation hdi at the d-th dimension is obtained from the final step hidden state h d,τ i via a linear transformation: hdi = Wo · h d,τ i + bo. Concatenating the path representations at all D dimensions and passing it through a fully-connected layer gives the final path embedding epathTi ∈ R de for task Ti: epathTi = FC([h 1 i , ...,h D i ] ⊤).
Inspired by (Suo et al., 2020; Dong et al., 2020; Lin et al., 2021), we handle task heterogeneity without jeopardizing generalization among similar tasks by employing a simple yet effective soft K-means clustering on the path embeddings2. Specifically, we maintain kpath learnable cluster centroids {apathj |∀j ∈ [1, kpath]} ∈ Rkpath×de for path embeddings. The cluster-enhanced path embedding ẽpathTi ∈ R de is simply the weighted sum of the cluster centroids:
ẽpathTi = ∑kpath j=1 qpathij a path j , (3)
where qpathij = (1+||epathTi −a path j || 2)−1
∑ j′ (1+||e path Ti −apath j′ ||
2)−1 is the probability of assigning epathTi to cluster j, computed
using the Student’s t -distribution as a kernel, following Xie et al. (2016).
4.2 TASK REPRESENTATION BASED ON FEATURES Task representation based on learning path can be interpreted as encoding the conditional distribution p(y|x) (as the learning path is informed by the labels), whereas task is best described by the joint distribution p(x, y) = p(y|x)p(x). Therefore, we further incorporate the marginal distribution of input features by learning another representation solely based on features, as what have been done in most of the existing task-aware meta-learning methods (Vuorio et al., 2019; Yao et al., 2019; 2020; Lin et al., 2021). Another significance of including feature-based representation is that, we can create a mapping between the path and feature representations to bypass the rehearsed learning during the meta-testing phase (details will be elaborated in Section 4.4).
Generally, the design of the feature extractor may vary for different application domains. Let E(·) denote an arbitrary feature extractor, the feature embedding efeatTi ∈ R de for task Ti is obtained by
aggregating the extracted features of all samples in the training set: efeatTi = 1
ntrTi
∑ntrTi j=1(E(xi,j)).
Similar to the path embeddings, we also employ a soft K-means clustering to promote generalization among related feature embeddings. Specifically, given kfeat cluster centroids {afeatj |∀j ∈
1Other network designs are possible for the meta path learner. We compare their efficacy in ablation study. 2Other clustering algorithms are possible. We leave it to future work.
[1, kfeat]} ∈ Rkfeat×de , the cluster-enhanced feature embedding ẽfeatTi ∈ R de is obtained by:
ẽfeatTi = ∑kfeat j=1 qfeatij a feat j , (4)
where qfeatij = (1+||efeatTi −a feat j || 2)−1
∑ j′ (1+||e feat Ti −afeat j′ ||
2)−1 is the probability of assigning efeatTi to cluster j.
4.3 TASK-AWARE MODULATION We aggregate the path and feature embeddings via a learnable weight vector λ to generate the final task representation. This task representation will then be used to produce a modulation to tailor the global initialization θ0 to specific task. The modulated initialization θ0i for task Ti is obtained by:
θ0i = σ(W · (λẽpathTi ⊕ (1− λ)ẽ feat Ti ) + b)⊙ θ0, (5)
where W, b, λ are learnable parameters, σ(·) is the sigmoid function, ⊕ and ⊙ denote element-wise addition and element-wise multiplication respectively.
From this modulated initialization, task Ti will undergo regular task adaptation learning for τ steps. After that, the meta-learner φ will be updated by optimizing loss across all the test sets, the same procedure as in MAML. The meta-optimization objective is:
min φ
∑
Ti∈Ttr L(fθ0i−α∑τ−1t=0 ▽θL(fθtTi ,D tr Ti ),D te Ti), (6)
where φ includes the global initialization θ0, the learnable parameters used to generate the clusterenhanced path and feature embeddings, and also the final modulation.
4.4 IMPROVING META-TESTING EFFICIENCY VIA SHORTCUT TUNNEL The need to conduct rehearsed learning before the actual learning of each task inevitably leads to twice the inference time compared to the original MAML. Though it is not possible to bypass the rehearsed learning during meta-training as we need to optimize the meta path learner and the downstream cluster centroids, it is possible to improve the inference efficiency during meta-testing with the well-trained cluster centroids. Note that the cluster-enhanced path embedding ẽpathTi used to generate modulation is obtained based on the path cluster centroids and the soft assignment (see equation 3). If we can estimate the path cluster assignment without actually going through the rehearsed learning process to obtain epathTi , we will be able to reduce the inference time by half.
Inspired by this, we devise a shortcut tunnel to predict the path cluster assignment directly from the feature cluster assignment of the same task. The assumption behind is that there exists a one-toone mapping between the feature cluster assignments and the path cluster assignments, which can be linear or non-linear. For better expressivity, we employ a two-layer fully connected network to approximate the mapping and use it to reconstruct the path cluster assignment of Ti from its feature cluster assignment:
q̂pathi = softmax(FCs(q feat i )), (7)
where qfeati = [q feat i1 , ..., q feat ikfeat ]⊤ ∈ Rkfeat and q̂pathi = [q̂ path i1 , ..., q̂ path ikpath ]⊤ ∈ Rkpath . The mapping handles cases where kpath ∕= kfeat and allows for different softness of cluster assignments. To align the reconstructed and the actual assignment distributions, we use the Jensen-Shannon (JS) divergence (a symmetrized version of Kullback-Leibler (KL) divergence) as the reconstruction loss:
Lr(DtrTi) = JS(q̂ path i ||q path i ) =
1 2 KL(q̂pathi ||pi) + 1 2 KL(qpathi ||pi), (8)
where pi = 12 (q̂ path i + q path i ), and KL(q||p) = ∑ j qj log qj pj
is the KL divergence. During metatraining, the reconstruction loss is optimized together with the loss in equation 6, resulting in the following overall objective:
min φ
∑
Ti∈Ttr L(fθ0i−α∑τ−1t=0 ▽θL(fθtTi ,D tr Ti ),D te Ti) + ζLr(D tr Ti), (9)
where ζ controls the weight of Lr(DtrTi), and φ now further includes the shortcut tunnel parameters.
To apply the shortcut tunnel during meta-testing, we simply replace qpathi with q̂ path i in equation 3 to obtain the cluster-enhanced path embedding ẽpath, i.e., ẽpathTi = ∑kpath j=1 q̂ path ij a path j . The overall meta-training and meta-testing procedures of CTML are summarized in Algorithm 1 and 2.
Algorithm 1: Meta-Training of CTML
Required: Meta-training tasks Ttr = {Ti}N tr
i=1 ; Number of steps τ ; Adaptation rate α; Meta-update rate β; Number of clusters kpath and kfeat
1 Randomly initialize all learnable parameters φ 2 while not converged do 3 Sample a batch of tasks B from Ttr 4 for Ti ∈ B do 5 Extract efeatTi and cluster to obtain ẽ feat Ti 6 Extract epathTi from the τ -step rehearsed learning and cluster to obtain ẽpathTi 7 Aggregate ẽfeatTi and ẽ path Ti to generate θ0i 8 Evaluate L(fθTi ,D te Ti) and Lr(D tr Ti) 9 Meta-update φ ← φ− β▽φ ∑ Ti∈B[L(fθTi ,D te Ti) + ζLr(D tr Ti)]
Algorithm 2: Meta-Testing of CTML Required: Meta-testing tasks Tte = {Ti}Ni=NtrTi+1 ;
Number of steps τ ; Adaptation rate α; Number of clusters kpath and kfeat; Meta-trained φ
1 for Ti ∈ Tte do 2 Extract efeatTi and cluster to obtain ẽ feat Ti 3 if use shortcut approximation then 4 Generate path assignment via the shortcut tunnel to obtain ẽpathTi 5 else 6 Extract epathTi from the τ -step rehearsed learning and cluster to obtain ẽpathTi 7 Aggregate ẽfeatTi and ẽ path Ti to generate θ0i 8 Evaluate L(fθTi ,D te Ti)
5 EXPERIMENTS
To test the effectiveness of the proposed CTML framework, we conduct experiments3 on two realworld application domains: few-shot image classification and cold-start recommendation.
5.1 FEW-SHOT IMAGE CLASSIFICATION
Datasets and Settings In few-shot image classification, each task is defined as assigning images to N classes after training with K samples (i.e., N -way K-shot) (Vinyals et al., 2016). To simulate task heterogeneity, we construct the Mixture-Of-Datasets consisting of 6 widely used benchmark datasets: CUB-200-2011 (Bird) (Wah et al., 2011), FGVC-Aircraft (Aircraft) (Maji et al., 2013), FGVCx-Fungi (Fungi) (Schroeder & Cui, 2018), VGG Flower (Flower) (Nilsback & Zisserman, 2008), Describable Textures (Texture) (Cimpoi et al., 2014), and GTSRB Traffic Signs (Traffic Sign) (Stallkamp et al., 2012). Following Yao et al. (2019), we create each task by sampling N classes from one of the 6 sub-datasets. This step ensures that tasks are drawn from different underlying distributions. Apart from the Mixture-Of-Datasets, we also conduct experiments on the MiniImagenet (Ravi & Larochelle, 2016), which can be considered as a task-homogeneous setting as tasks are sampled from a single dataset (results on MiniImagenet are discussed in Appendix D.2). More details on datasets and pre-processing can be found in Appendix A.1.
Baselines and Our Method We compare the performance of our method against 5 baselines: (1) global initialization method: MAML (Finn et al., 2017); (2) task-specific initialization methods: MMAML (feature-based customization) (Vuorio et al., 2019) and L2F (gradient-based customization) (Baik et al., 2020); (3) cluster-enhanced initialization methods: HSML (feature-based hierarchical clustering) (Yao et al., 2019) and ARML (feature-based relational graph) (Yao et al., 2020). For our proposed CTML, we report both the original meta-testing performance (CTML) and the one applying shortcut approximation (CTML(approx.)). We further implement 2 variants to compare with the baselines: CTML-feat which only depends on feature representation and CTML-path which only depends on path representation. Following Finn et al. (2017), we adopt both the feature extractor E(·) and the base-learner fθ as a four-layer 3 × 3 convolutions with 32 filters for all the compared methods (experiments on deeper network for base-learner can be found in Appendix D.1). Further details on hyper-parameters settings can be found in Appendix B.1.
Baseline Comparison Table 1a presents the 5-way 1-shot & 5-shot classification performance on Mixture-Of-Datasets (disentangled performance on individual sub-datasets are included in Appendix D.3). First of all, we see that all the task-aware methods outperform the globally initialized MAML in task-heterogeneous setting. L2F with gradient-based conditioning on the initialization performs better than the feature-based MMAML. After considering generalization across similar tasks with some clustering techniques, HSML and ARML further improve over MMAML. Under our proposed CTML framework, CTML-feat with K-means clustering on features exhibits comparable performance as HSML and ARML. For CTML-path, the improvements over baselines are more
3All experiments are conducted using NVIDIA Tesla P100 GPU with 16GB memory.
significant, especially for the 5-shot scenario where the learning paths are more reliable with more training data. The final CTML combining the advantages of the two representations achieves further improvements and outperforms the strongest baseline by a significant margin. Nevertheless, we can see that the meta-testing time of CTML is about twice that of the baselines due to the additional rehearsed learning process. After applying the shortcut approximation, we are able to cut down the inference time by half with just a small compromise on performance. Ablation Study We further conduct ablation study to test the efficacy of various CTML designs, including the step-wise input components, the path learner design, and the effects of clustering. From the results in Table 1b, we see that the lower-order quantities seem more important. Regarding the path learner design, our proposed GRU-based model achieves the best result due to its ability to capture the sequential dependencies among the steps. Attention model is also a competitive alternative, as the positional encodings and the causal mask serve to constrain the sequential order (detailed descriptions of the different path learner designs can be found in Appendix C). As for clustering, we see that it indeed helps to generalize better as the global structure is explicitly considered. Understanding the Path Learning Mechanism To understand what has actually been learned by the GRU path learner, we randomly sample 6 tasks (5-way 1-shot) from Aircraft, Flower and Traffic Sign sub-datasets (Figure 3a) and visualize their path learning results. In Figure 3b, we plot the 5- step rehearsed learning paths (upper) and the corresponding path embeddings generated by the GRU (lower) on the first two PCA components. We can see that the learning paths of tasks from the same sub-dataset generally move along the same direction, and the corresponding path embeddings are able to encode the path geometric information and positioned closely for similar tasks in the latent space. In Figure 3c, we further visulize the z gates and r gates of the GRU at each step t, which control how much to retain for the current step input and previous step memory respectively. We can see that the GRU path learner is able to identify the “important” steps (darker-blue blocks) and absorbs a larger portion of the inputs at these steps into the final path embedding. More examples on path visualization can be found in Appendix D.5
5.2 COLD-START RECOMMENDATION Datasets and Settings In the field of recommender systems, meta-learning has been applied to solve the cold-start problem, which refers to making recommendations related to new items or new users unseen before (Vartak et al., 2017; Volkovs et al., 2017; Lee et al., 2019). Here, we focus on the user cold-start problem, where recommending for each user is treated as a task. This setting is task-heterogeneous in nature as users may belong to different preference groups. We conduct experiments on 3 public datasets: MovieLens-1M (Harper & Konstan, 2015), Yelp (Dataset, 2019), and Amazon-CDs (McAuley et al., 2015), consisting of user’s ratings on movies, business services and CD products respectively. We split each dataset into meta-train/val/test by ratio 7/1/2 according to the timestamps of ratings. For each user, we use the first 10 samples as training set, and the rest as test set. Further details about datasets and pre-processing can be found in Appendix A.2. Baselines and Our Method We choose 4 baselines that specifically tackle the user cold-start problem in recommender systems developed based on MAML. MeLU (Lee et al., 2019) is the first to adopt MAML in recommender systems with locally adapted decision layers; MetaHIN (Lu et al., 2020) further incorporates Heterogeneous Information Networks (HIN) for data augmentation and multi-faceted adaptations; MAMO (Dong et al., 2020) involves a global memory module to group users based on profile information and personalizes the initial bias; PAML (Yu et al., 2021) personalizes the adaptation learning rate to allow for better fitting of the minor users. Following the baselines, we implement the feature extractor E(·) as several embedding lookup matrices for different features, and the base-learner fθ as a general recommender system model consists of embedding matrices followed by multiple fully-connected layers (Lee et al., 2019). Detailed hyper-parameters settings can be found in Appendix B.2. Baseline Comparison We evaluate the rating prediction performance in terms of mean absolute error (MAE) on 3 datasets, reported in Table 2 (more detailed results can be found in Appendix D.4). From the results, we can see that PAML and MAMO with personalized adaptation rate and initialization achieve better performance than MELU. Our CTML surpasses all the baselines, including MetaHIN which leverages HIN for data augmentation. Furthermore, we notice that for MovieLens, CTML-feat is slightly better than CTML-path, while for the other two datasets, CTML-path significantly outperforms CTML-feat. Given that the MovieLens dataset contains more complete user profile, this result implies the potential of representing users based on learning paths when the side information is seriously lacking or inaccurate, which is often the case in real-world applications. Number of Clusters For recommendation problem, there is no ground-truth grouping of users. In Figure 4a, we investigate the effect of varying the number of clusters kpath and kfeat for MovieLens dataset. We see that the best performance occurs at smaller kfeat values and larger kpath values, which implies greater complexity for clustering the learning paths. Our design of the shortcut tunnel allows flexibility to set different kpath and kfeat. In Figure 4b, we visualize the shortcut approximation of path assignment from feature assignment for a random user of MovieLens. We can see that the shortcut tunnel is rather reliable as the reconstructed path assignment highly resembles the actual path assignment. More examples on shortcut approximation are provided in Appendix D.6.
6 CONCLUSION In this work, we introduce a CTML framework which leverages both features and learning paths for task representation. We employ a GRU-based meta path learner to process the step-wise geometric quantities, and introduce a shortcut tunnel to bypass the rehearsed learning during meta-testing. Experiments on two real-world application domains demonstrate the effectiveness of our framework.
7 REPRODUCIBILITY STATEMENT
Our code can be accessed at:
https://github.com/didiya0825/ctml_code
For data pre-processing and hyper-parameters settings, please see Appendix A and B.
8 ETHICS STATEMENT
Since the proposed method is a general meta-learning framework, we do not foresee any potential ethic issue to the best our knowledge.
REFERENCES
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6430–6439, 2019.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 139–154, 2018.
Sungyong Baik, Seokil Hong, and Kyoung Mu Lee. Learning to forget for meta-learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2379– 2387, 2020.
Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, and Leonid Sigal. Improved fewshot visual classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14493–14502, 2020.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. In International Conference on Learning Representations, 2018.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3606–3613, 2014.
Yelp Dataset. Yelp dataset challenge, 2019.
Arnout Devos and Yatin Dandi. Model-agnostic learning to meta-learn. arXiv preprint arXiv:2012.02684, 2020.
Manqing Dong, Feng Yuan, Lina Yao, Xiwei Xu, and Liming Zhu. Mamo: Memory-augmented meta-optimization for cold-start recommendation. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 688–697, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning, pp. 1126–1135. PMLR, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 9537–9548, 2018.
Sebastian Flennerhag, Pablo G Moreno, Neil D Lawrence, and Andreas Damianou. Transferring knowledge across learning processes. In International Conference on Learning Representations, 2018.
Jezabel R Garcia, Federica Freddi, Feng-Ting Liao, Jamie McGowan, Tim Nieradzik, Da-shan Shiu, Ye Tian, and Alberto Bernacchia. Meta-learning with maml on trees. arXiv preprint arXiv:2103.04691, 2021.
Francesco Gianfelici and Viviana Battistelli. Methods of information geometry (amari, s. and nagaoka, h.; 2000)[book review]. IEEE Transactions on Information Theory, 55(6):2905–2906, 2009.
F Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis), 5(4):1–19, 2015.
Sameeksha Katoch, Kowshik Thopalli, Jayaraman J Thiagarajan, Pavan Turaga, and Andreas Spanias. Invenio: Discovering hidden relationships between tasks/domains using structured meta learning. arXiv preprint arXiv:1911.10600, 2019.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
Hoyeop Lee, Jinbae Im, Seongwon Jang, Hyunsouk Cho, and Sehee Chung. Melu: meta-learned user preference estimator for cold-start recommendation. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1073–1082, 2019.
Huaiyu Li, Weiming Dong, Xing Mei, Chongyang Ma, Feiyue Huang, and Bao-Gang Hu. Lgm-net: Learning to generate matching networks for few-shot learning. In International conference on machine learning, pp. 3825–3834. PMLR, 2019.
Xixun Lin, Jia Wu, Chuan Zhou, Shirui Pan, Yanan Cao, and Bin Wang. Task-adaptive neural process for user cold-start recommendation. arXiv preprint arXiv:2103.06137, 2021.
Yuanfu Lu, Yuan Fang, and Chuan Shi. Meta-learning on heterogeneous information networks for cold-start recommendation. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1563–1573, 2020.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013.
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image-based recommendations on styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pp. 43–52, 2015.
Tsendsuren Munkhdalai and Hong Yu. Meta networks. In International Conference on Machine Learning, pp. 2554–2563. PMLR, 2017.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722–729. IEEE, 2008.
Boris N Oreshkin, Pau Rodrı́guez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In NeurIPS, 2018.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dÁlché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 7957–7968. Curran Associates, Inc., 2019.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In International Conference on Learning Representations, 2018.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In International Conference on Learning Representations, 2018.
B Schroeder and Y Cui. Fgvcx fungi classification challenge 2018, 2018.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, 2017.
Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural networks, 32:323–332, 2012.
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 403–412, 2019.
Qiuling Suo, Jingyuan Chou, Weida Zhong, and Aidong Zhang. Tadanet: Task-adaptive network for graph-enriched meta-learning. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1789–1799, 2020.
Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. In Learning to learn, pp. 3–17. Springer, 1998.
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: a good embedding is all you need? In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16, pp. 266–282. Springer, 2020.
Manasi Vartak, Arvind Thiagarajan, Conrado Miranda, Jeshua Bratman, and Hugo Larochelle. A meta-learning perspective on cold-start recommendations for items. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 6907–6917, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998–6008, 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 3637–3645, 2016.
Maksims Volkovs, Guangwei Yu, and Tomi Poutanen. Dropoutnet: addressing cold start in recommender systems. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 4964–4973, 2017.
Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph J. Lim. Multimodal model-agnostic metalearning via task-aware modulation. In Neural Information Processing Systems, 2019.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International conference on machine learning, pp. 478–487. PMLR, 2016.
Huaxiu Yao, Ying Wei, Junzhou Huang, and Zhenhui Li. Hierarchically structured meta-learning. In International Conference on Machine Learning, pp. 7045–7054. PMLR, 2019.
Huaxiu Yao, Xian Wu, Zhiqiang Tao, Yaliang Li, Bolin Ding, Ruirui Li, and Zhenhui Li. Automated relational meta-learning. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=rklp93EtwH.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 7343–7353, 2018.
Runsheng Yu, Yu Gong, Xu He, Yu Zhu, Qingwen Liu, Wenwu Ou, and Bo An. Personalized adaptive meta learning for cold-start user preference prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 10772–10780, 2021.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. arXiv preprint arXiv:2001.06782, 2020.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pp. 3987–3995. PMLR, 2017.
A DATASETS AND PRE-PROCESSING DETAILS
If not specially mentioned, datasets are downloaded from the respective official websites, where we have tried our best to seek permission for usage if applicable. The open-source datasets are licensed under CC0: Public Domain. For recommendation datasets, all records are anonymized and no personal identifiable information is involved.
A.1 IMAGE DATASETS
In our experiments, Mixture-Of-Datasets consist of six widely used benchmark datasets. For the CUB-200-2011 (Bird), FGVC-Aircraft (Aircraft), FGVCx-Fungi (Fungi), and DTD (Texture) datasets, we directly acquire the pre-processed version released by the authors of HSML4 (Yao et al., 2019). Pre-processing details of these four sub-datasets can be found in the original paper of HSML. We have obtained the authors’ consent to use this asset for our research.
For the VGG Flower (Flower)5 and GTSRB (Traffic Sign)6 datasets, we download the raw datasets from the respective official websites. We follow the pre-processing protocols of Finn et al. (2017) to resize all the images to 84×84×3. We also try to maintain the same meta-train/validation/test ratio as Yao et al. (2019) for class splits. For VGG Flower, there are in total 102 flower classes, and each class consists of 40 to 258 images. For GTSRB, we use the official training data which contains 43 traffic sign classes, and each class has 30 images.
For MiniImagenet, we acquire the dataset as well as the meta-train/validation/test splits from Ravi & Larochelle (2016). Table 3 summarizes the meta-train/validation/test splits for each of the subdatasets as well as MiniImagenet.
A.2 RECOMMENDATION DATASETS
A.2.1 MOVIELENS-1M
MovieLens-1M7 dataset (Harper & Konstan, 2015) contains 1,000,209 anonymous ratings on around 3,900 movies made by 6,040 users. We acquire the additional side information released by the authors of MeLU8 (Lee et al., 2019), which contains user profile information of gender, age group, occupation, zipcode, and movie profile information of rate, genre, director and actor.
The dataset is split into meta-train/validation/test set according to the timestamps of ratings by ratio 7/1/2. As a result, the start time is 2000-04-26, the meta-train cut-off time is 2000-11-22, the metavalidation cut-off time is 2000-12-02, and the end time is 2003-03-01. Users with less than 30 samples in each split are removed. In the end, the average sample size for each user is 163. The statistics of the pre-processed MovieLens-1M dataset and the number of meta-train/validation/test users are recorded in Table 4 and 5. Note that it is possible for meta-train users to appear in metavalidation and meta-test set, and these users are termed warm users. Conversely, meta-validation or meta-test users that have never appeared in the meta-train set are termed cold users.
4https://github.com/huaxiuyao/HSML 5https://www.robots.ox.ac.uk/˜vgg/data/flowers/ 6https://benchmark.ini.rub.de/ 7https://grouplens.org/datasets/movielens/ 8https://github.com/hoyeoplee/MeLU
A.2.2 YELP
Yelp9 dataset (Dataset, 2019) contains 8,635,403 reviews for 160,585 business services made by 2,189,457 users. We extract the star ratings from the reviews for our experiments. For each business service, we collect the city it resides and the categories it belongs to from the business metadata. However, for the user, the personal information is lacking, and we only have access to the user ID.
The dataset is split into meta-train/validation/test set according to the timestamps of ratings by ratio 7/1/2. As a result, the start time is 2004-10-12, the meta-train cut-off time is 2018-02-07, the metavalidation cut-off time is 2018-09-04, and the end time is 2019-12-13. Users with less than 30 samples in each split are removed. In the end, the average sample size for each user is 89. The statistics of the pre-processed Yelp dataset and the number of meta-train/validation/test users are recorded in Table 4 and 5.
A.2.3 AMAZON-CDS
Amazon-CDs10 dataset (McAuley et al., 2015) contains 3,749,004 ratings for 544,442 CD and Vinyl products made by 1,578,597 users. For each CD, we collect its brand and category, while for the user, the personal profile information is lacking and we only have access to the user ID.
The dataset is split into meta-train/validation/test set according to the timestamps of ratings by ratio 7/1/2. As a result, the start time is 1997-09-13, the meta-train cut-off time is 2012-01-13, the metavalidation cut-off time is 2013-03-05, and the end time is 2014-07-23. Users with less than 30 samples in each split are removed. In the end, the average sample size for each user is 82. The statistics of the pre-processed Amazon-CDs dataset and the number of meta-train/validation/test users are recorded in Table 4 and 5.
B HYPER-PARAMETERS SETTINGS
B.1 FEW-SHOT IMAGE CLASSIFICATION
For fair comparison, we adopt the same feature extractor and base-learner for all the compared methods. Following Finn et al. (2017), the feature extractor is a four-layer 3 × 3 convolutions with 32 filters, followed by a one-layer fully-connected layer that maps the extracted features into the task embedding dimension de. The base-learner is also a four-layer 3 × 3 convolutions with 32 filters, followed by a one-layer fully-connected layer that projects the output to prediction scores for 5 classes. Each convolution layer is accompanied with batch normalization, ReLU nonlinearity, and 2× 2 max-pooling. For all the compared methods, we follow the hyper-parameters settings for meta-training in MAML (Finn et al., 2017), which sets the adaptation learning rate α as 0.01, the meta-update learning rate β as 0.001, the number of adaptation steps τ as 5, the meta batch size |B| as 4, the size of test set |DteTi | as 15, and the number of meta-update iterations as 60,000. For all the task-aware modulation methods (i.e., MMAML (Vuorio et al., 2019), L2F (Baik et al., 2020), HSML (Yao et al., 2019), ARML (Yao et al., 2020)), we set the dimension of task embedding de as 128 (following Yao et al. (2019)). Additionally, for HSML (Yao et al., 2019), we set the number of clusters in the first and second layer of hierarchical tree as 4 and 2 respectively. For ARML (Yao et al., 2020), we set the number of vertices in the meta-knowledge graph as 4. For our proposed CTML, we set the number of clusters for both feature and path representations as 6, i.e., kpath = kfeat = 6.
B.2 COLD-START RECOMMENDATION
Different from the image dataset, recommendation dataset typically consists of user-item interactions and profile features for both sides. The feature extractor is implemented as several embedding lookup tables, each corresponds to one categorical feature. We set the embedding dimension as 8 for all features. The base-learner is implemented as a general deep learning-based recommender system model, composed of embedding matrices followed by a two-layer fully connected layer that models the interaction between different features. The embedding dimension for base-learner is set to 8, and the hidden sizes for the fully-connected layers is set to [32, 16].
For all the compared methods, we follow the hyper-parameters settings for meta-training in MeLU (Lee et al., 2019), which sets the adaptation learning rate α as 5e-3, the meta-update learning rate β as 5e-5, the number of adaptation steps τ as 5. For other hyper-parameters that are not explicitly specified by Lee et al. (2019), we set the meta batch size |B| as 1, the size of training set |DtrTi | for each user as 10 (the remaining are used as test set DteTi for that user, whose size varies for different users), and the number of meta-training epochs as 20. For MAMO (Dong et al., 2020) which also employs K-means clustering, we apply the same hyper-parameters tuning procedure as our CTML. That is, we tune both kpath and kfeat in {4, 8, 12, 16, 20}, and tune the dimension of task embedding de in {64, 128, 256, 512, 1024}. Other hyper-parameters settings specific to the baseline methods are directly adopted from the source codes released by the authors.
C OTHER PATH LEARNER DESIGNS
In addition to the GRU-based meta path learner design described in equation 2, we also experiment on 3 other network designs to test their effectiveness in modeling the step-wise geometric quantities.
Recall that the meta path learner is applied coordinate-wise on the base-learner parameters. That is, the same meta path learner is used to process the sequence of step-wise tuples PdTi ∈ R (τ+1)×4 independently for all parameters at different dimensions d ∈ {1, 2, ..., D}. The first design linearly transforms the vectorized PdTi with learnable weight and bias:
Linear : hdi = W · vec(PdTi) + b. (10)
The second one employs multiple fully-connected layers with nonlinear activation σ(·):
FC : u0 = vec(PdTi),
u1 = σ(W1 · u0 + b1), ...
uL = σ(WL · uL−1 + bL), hdi = u L.
(11)
The third design applies multiple self-attention layers on the sequence with scaled dot-product similarity following Vaswani et al. (2017). To inform about the sequential order, we apply sinusoid positional encodings at the input level, and also apply causal mask to prevent interaction with future steps (i.e., set the upper triangular entries of the similarity matrix to −∞ before inputting into softmax). The final representation hdi is given by the (τ + 1)-th row (i.e., the last row) of the final-layer output:
Attention : U0 = PdTi + positional encodings,
U1 = softmax( Q1 · (K1)⊤√
da ) ·V1,
...
UL = softmax( QL · (KL)⊤√
da ) ·VL,
hdi = U L τ+1,:,
where Ql = σ(Ul−1 ·WlQ + blQ) ∈ R(τ+1)×da , Kl = σ(Ul−1 ·WlK + blK) ∈ R(τ+1)×da , Vl = σ(Ul−1 ·WlV + blV ) ∈ R(τ+1)×da .
(12)
Concatenating the path representations at all D dimensions and passing it through a fully-connected layer gives the final path embedding epathTi ∈ R de for task Ti:
epathTi = FC([h 1 i , ...,h D i ] ⊤). (13)
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 PERFORMANCE ON RESNET-12
We have conducted additional experiments to test the effectiveness of our method using a deeper network ResNet-12 as the backbone (i.e., the base-learner). We follow the implementation of Sun et al. (2019) which adopts (64-128-256-512) for the number of filters at the four blocks.
It was shown recently that, with a deeper backbone, simple training paradigm like pre-training and fine-tuning (Tian et al., 2020; Chen et al., 2018) can achieve very satisfactory performance. We therefore further include the following 3 baselines for comparison on ResNet-12:
• Finetune: This baseline simply pretrains a global feature extractor on the entire training dataset and then finetunes the classifier for individual tasks.
• Finetune-distance (Chen et al., 2018): This method modifies Finetune by computing cosine distance between the input feature vector and the class vectors for the classifier.
• Finetune-distill (Tian et al., 2020): This method modifies Finetune by employing a sequential self-distillation technique to pretrain the feature extractor.
Table 6 below presents the 5-way 1-shot performance on the Mixture-Of-Datasets with 2 different backbones: Conv-4 and ResNet-12. We can see that the gaps between Finetune methods and MAML are narrower for deeper backbone, which can be attributed to that the deeper models have larger capacity to accommodate the transferable knowledge obtained from the pre-training (in line with the findings of Chen et al. (2018)). However, under this task-heterogeneous setting, the taskconditioned methods still yield better performance than MAML and the Finetune baselines for ResNet-12, demonstrating the effectiveness of task-conditioning even for deeper backbone. And our method with path representation yields the best result.
D.2 PERFORMANCE ON MINIIMAGENET
Table 7 presents the results on MiniImagenet benchmark (Ravi & Larochelle, 2016), where tasks are constructed from a single dataset (i.e., there is only one underlying distribution) and hence can be considered as a task-homogeneous setting. Since in this case, task-aware methods do not have advantages over the global initialization methods, we see that all the baselines have comparable performance as the original MAML. Nevertheless, the results show that though the task-aware modulation is rendered less meaningful in task-homogeneous setting, CTML will yield performance at least comparable to the global initialization method in this scenario.
D.3 PERFORMANCE ON SUB-DATASETS
Table 8 presents the few-shot classification performance of CTML and the baselines on each subdataset. We see that CTML consistently outperforms all the baselines for all the sub-datasets. Other trends are similar to that of the overall results in Table 1a.
D.4 PERFORMANCE ON RECOMMENDATION DATASETS
Table 9 presents the rating prediction performance on 3 datasets evaluated in two metrics: mean absolute error (MAE) and normalized discounted cumulative gain at top 20 (NDCG@20). The meta-testing users are further divided into warm and cold users, where warm users are those that have appeared in the meta-training set, and cold users are those that have not11. From the results, we see that CTML consistently outperforms the baselines.
11Note that for few-shot image classification, all the meta-testing tasks are unseen during meta-training, while for recommendation, it is possible that the meta-testing users also appeared in meta-training set.
D.5 ADDITIONAL LEARNING PATH VISUALIZATION
D.5.1 FEW-SHOT IMAGE CLASSIFICATION
Here we provide more examples for visualizing the path learning results for the few-shot image classification experiments. Figure 5, 6 and 7 each shows the path learning results of 6 tasks randomly sampled from 3 of the 6 sub-datasets.
We see that in this application scenario, the overall direction of the learning path plays an important role in determining the task representation. Looking at the GRU gates at each step, it seems that the first 2 to 3 steps are often the most important/informative steps, after which the learning begins to converge and the subsequent steps no longer provide useful information about the learning direction.
D.5.2 COLD-START RECOMMENDATION
We also provide examples for visualizing the learning paths for the cold-start recommendation experiments. Figure 8 and 9 each shows the path learning results of 6 users randomly selected from the MovieLens-1M dataset, and Figure 10 and 11 each shows the results of 6 users sampled from the Amazon-CDs dataset.
In this application scenario, we notice that apart from the overall direction of the path, the length of the path also plays an important role in determining the task representation. For instance, in Figure 11, the blue, orange, green and brown paths (upper plot) are clustered closely in the representation space (lower plot) due to similar lengths, although they are pointing at very different directions. Similar observation can also be found in Figure 8. Although blue and orange paths are pointing at the same directions, their embeddings are positioned far away from each other due to the big difference in path lengths.
D.6 ADDITIONAL SHORTCUT APPROXIMATION VISUALIZATION
In this section, we give more examples to verify the effectiveness of the shortcut approximation. Figure 12 shows the approximation results of 2 users sampled from the MovieLens-1M dataset, and Figure 13 shows the results of 3 tasks sampled from 3 different image sub-datasets. We can see that the shortcut tunnel is able to learn and accommodate different mappings, provide guarantee on the performance of the approximated model.