summaries: The paper proposes a few-shot meta-learning method for recommender system that uses a new feature's meta-information and observed samples for the features to predict the network weights for predicting the feature value from other features. The paper focuses on the cold-start problem where few samples with a new feature observed is available. The method outperforms a wide range of baselines on MovieLens-1M, a medical synthetic dataset, and a e-learning dataset. The paper proposes Contextual HyperNetworks (CHNs) as an auxiliary model to generate parameters from existing data, and observations and other metadata associated with new feature to address cold start problem of new feature. Besides, it doesn’t need either re-train or fine-tune at prediction time. The CHN is applied to P-VAE and some experimental results are provided to demonstrate its effectiveness in some application, i.e., recommender system, e-learning and healthcare tasks. This submission focuses on the cold start problem of new entities (new items in a recommender system, new treatments in a medical application, etc.). It combines the strengths of the *relations* between a new entity and the existing entities, and the *content* features of the new entity, by fusing the two kinds of information into a neural network that outputs the estimated representation of the new entity. The proposed method outperforms several intuitive naïve strategies as well as MAML.
abstractText: While deep learning has obtained state-of-the-art results in many applications, the adaptation of neural network architectures to incorporate new output features remains a challenge, as a neural networks are commonly trained to produce a fixed output dimension. This issue is particularly severe in online learning settings, where new output features, such as items in a recommender system, are added continually with few or no associated observations. As such, methods for adapting neural networks to novel features which are both time and data-efficient are desired. To address this, we propose the Contextual HyperNetwork (CHN), an auxiliary model which generates parameters for extending the base model to a new feature, by utilizing both existing data as well as any observations and/or metadata associated with the new feature. At prediction time, the CHN requires only a single forward pass through a neural network, yielding a significant speed-up when compared to re-training and fine-tuning approaches. To assess the performance of CHNs, we use a CHN to augment a partial variational autoencoder (P-VAE), a deep generative model which can impute the values of missing features in sparsely-observed data. We show that this system obtains improved few-shot learning performance for novel features over existing imputation and meta-learning baselines across recommender systems, e-learning, and healthcare tasks.
references.authors: Marko Balabanović Yoav Shoham
references.title: Fab: content-based, collaborative recommendation
references.venue: Communications of the ACM,
references.authors: Luca Bertinetto João F Henriques Jack Valmadre Philip Torr Andrea Vedaldi
references.title: Learning feedforward one-shot learners
references.venue: In Advances in neural information processing systems,
references.authors: Homanga Bharadhwaj
references.title: Meta-learning for user cold-start recommendation
references.venue: International Joint Conference on Neural Networks (IJCNN),
references.authors: Chelsea Finn Pieter Abbeel Sergey Levine
references.title: Model-agnostic meta-learning for fast adaptation of deep networks
references.venue: In Proceedings of the 34th International Conference on Machine Learning-Volume
references.authors: Marta Garnelo Dan Rosenbaum Chris J Maddison Tiago Ramalho David Saxton Murray Shanahan Yee Whye Teh Danilo J Rezende SM Eslami
references.title: Conditional neural processes
references.venue: arXiv preprint arXiv:1807.01613,
references.authors: Spyros Gidaris Nikos Komodakis
references.title: Dynamic few-shot visual learning without forgetting
references.venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
references.authors: Carlos A Gomez-Uribe Neil Hunt
references.title: The netflix recommender system: Algorithms, business value, and innovation
references.venue: ACM Transactions on Management Information Systems (TMIS),
references.authors: Wenbo Gong Sebastian Tschiatschek Sebastian Nowozin Richard E Turner José Miguel HernándezLobato Cheng Zhang
references.title: Icebreaker: Element-wise efficient information acquisition with a bayesian deep latent gaussian model
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Jonathan Gordon John Bronskill Matthias Bauer Sebastian Nowozin Richard E Turner
references.title: Versa: Versatile and efficient few-shot learning
references.venue: In Third workshop on Bayesian Deep Learning,
references.authors: Edward Grefenstette Brandon Amos Denis Yarats Phu Mon Htut Artem Molchanov Franziska Meier Douwe Kiela Kyunghyun Cho Soumith Chintala
references.title: Generalized inner loop meta-learning
references.authors: Will Hamilton Zhitao Ying Jure Leskovec
references.title: Inductive representation learning on large graphs
references.venue: In Advances in neural information processing systems,
references.authors: F Maxwell Harper Joseph A Konstan
references.title: The movielens datasets: History and context
references.venue: Acm transactions on interactive intelligent systems (tiis),
references.authors: Xiangnan He Lizi Liao Hanwang Zhang Liqiang Nie Xia Hu Tat-Seng Chua
references.title: Neural collaborative filtering
references.venue: In Proceedings of the 26th international conference on world wide web,
references.authors: Diederik P Kingma Jimmy Ba
references.title: Adam: A method for stochastic optimization
references.venue: arXiv preprint arXiv:1412.6980,
references.authors: James Kirkpatrick Razvan Pascanu Neil Rabinowitz Joel Veness Guillaume Desjardins Andrei A Rusu Kieran Milan John Quan Tiago Ramalho Agnieszka Grabska-Barwinska
references.title: Overcoming catastrophic forgetting in neural networks
references.venue: Proceedings of the national academy of sciences,
references.authors: Xuan Nhat Lam Thuc Vu Trong Duc Le Anh Duc Duong
references.title: Addressing cold-start problem in recommendation systems
references.venue: In Proceedings of the 2nd international conference on Ubiquitous information management and communication,
references.authors: Dawen Liang Rahul G Krishnan Matthew D Hoffman Tony Jebara
references.title: Variational autoencoders for collaborative filtering
references.venue: In Proceedings of the 2018 World Wide Web Conference,
references.authors: Blerina Lika Kostas Kolomvatsos Stathes Hadjiefthymiades
references.title: Facing the cold start problem in recommender systems
references.venue: Expert Systems with Applications,
references.authors: Pasquale Lops Marco De Gemmis Giovanni Semeraro
references.title: Content-based recommender systems: State of the art and trends
references.venue: In Recommender systems handbook,
references.authors: Chao Ma Wenbo Gong José Miguel Hernández-Lobato Noam Koenigstein Sebastian Nowozin Cheng Zhang
references.title: Partial vae for hybrid recommender system
references.venue: In NIPS Workshop on Bayesian Deep Learning,
references.authors: Chao Ma Sebastian Tschiatschek Konstantina Palla José Miguel Hernández-Lobato Sebastian Nowozin Cheng Zhang
references.title: Eddi: Efficient dynamic discovery of high-value information with partial vae
references.venue: arXiv preprint arXiv:1809.11142,
references.authors: Tsendsuren Munkhdalai Xingdi Yuan Soroush Mehri Adam Trischler
references.title: Rapid adaptation with conditionally shifted neurons
references.venue: arXiv preprint arXiv:1712.09926,
references.authors: Cuong V Nguyen Yingzhen Li Thang D Bui Richard E Turner
references.title: Variational continual learning
references.venue: arXiv preprint arXiv:1710.10628,
references.authors: Alex Nichol John Schulman
references.title: Reptile: a scalable metalearning algorithm
references.venue: arXiv preprint arXiv:1803.02999,
references.authors: Adam Paszke Sam Gross Soumith Chintala Gregory Chanan Edward Yang Zachary DeVito Zeming Lin Alban Desmaison Luca Antiga Adam Lerer
references.title: Automatic differentiation in pytorch
references.authors: Michael J Pazzani Daniel Billsus
references.title: Content-based recommendation systems
references.venue: In The adaptive web,
references.authors: Charles R Qi Hao Su Kaichun Mo Leonidas J Guibas
references.title: Pointnet: Deep learning on point sets for 3d classification and segmentation
references.venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,
references.authors: James Requeima Jonathan Gordon John Bronskill Sebastian Nowozin Richard E Turner
references.title: Fast and flexible multi-task classification using conditional neural adaptive processes
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Badrul Sarwar George Karypis Joseph Konstan John Riedl
references.title: Item-based collaborative filtering recommendation algorithms
references.venue: In Proceedings of the 10th international conference on World Wide Web,
references.authors: J Ben Schafer Dan Frankowski Jon Herlocker Shilad Sen
references.title: Collaborative filtering recommender systems
references.venue: In The adaptive web,
references.authors: Andrew I Schein Alexandrin Popescul Lyle H Ungar David M Pennock
references.title: Methods and metrics for cold-start recommendations
references.venue: In Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval,
references.authors: Michael Schlichtkrull Thomas N Kipf Peter Bloem Rianne Van Den Berg Ivan Titov Max Welling
references.title: Modeling relational data with graph convolutional networks
references.venue: In European Semantic Web Conference,
references.authors: Suvash Sedhain Aditya Krishna Menon Scott Sanner Lexing Xie
references.title: Autorec: Autoencoders meet collaborative filtering
references.venue: In Proceedings of the 24th international conference on World Wide Web,
references.authors: Jake Snell Kevin Swersky Richard Zemel
references.title: Prototypical networks for few-shot learning
references.venue: In Advances in neural information processing systems,
references.authors: David H Stern Ralf Herbrich Thore Graepel
references.title: Matchbox: large scale online bayesian recommendations
references.venue: In Proceedings of the 18th international conference on World wide web,
references.authors: Ruibo Tu Kun Zhang Bo Bertilson Hedvig Kjellstrom Cheng Zhang
references.title: Neuropathic pain diagnosis simulator for causal discovery algorithm evaluation
references.venue: In Advances in Neural Information Processing Systems,
references.authors: Manasi Vartak Arvind Thiagarajan Conrado Miranda Jeshua Bratman Hugo Larochelle
references.title: A meta-learning perspective on cold-start recommendations for items
references.venue: In Advances in neural information processing systems,
references.authors: Oriol Vinyals Charles Blundell Timothy Lillicrap Daan Wierstra
references.title: Matching networks for one shot learning
references.venue: In Advances in neural information processing systems,
references.authors: Johannes von Oswald Christian Henning João Sacramento Benjamin F Grewe
references.title: Continual learning with hypernetworks
references.venue: arXiv preprint arXiv:1906.00695,
references.authors: Xiang Wang Xiangnan He Meng Wang Fuli Feng Tat-Seng Chua
references.title: Neural graph collaborative filtering
references.venue: In Proceedings of the 42nd international ACM SIGIR conference on Research and development in Information Retrieval,
references.authors: Zichao Wang Angus Lamb Evgeny Saveliev Pashmina Cameron Yordan Zaykov José Miguel Hernández-Lobato Richard E Turner Richard G Baraniuk Craig Barton Simon Peyton Jones
references.title: Diagnostic questions: The neurips 2020 education challenge
references.venue: arXiv preprint arXiv:2007.12061,
references.authors: Jiaxuan You Xiaobai Ma Yi Ding Mykel J Kochenderfer Jure Leskovec
references.title: Handling missing data with graph representation learning
references.venue: Advances in Neural Information Processing Systems,
references.authors: Qi
references.title: 2017), allowing it to efficiently encode and reconstruct partially observed data points. The P-VAE is based on the observation that typically the features in a VAE are assumed to be conditionally independent when conditioned on the latent variable z
references.venue: That is,
sections.heading: 1 INTRODUCTION
sections.text: In many deep learning application domains, it is common to see the set of predictions made by a model grow over time: a new item may be introduced into a recommender system, a new question may be added to a survey, or a new disease may require diagnosis. In such settings, it is valuable to be able to accurately predict the values that this feature takes within data points for which it is unobserved: for example, predicting whether a user will enjoy a new movie in a recommender system, or predicting how a user will answer a new question in a questionnaire.
  On the introduction of a new feature, there may be few or no labelled data points containing observed values for it; a newly added movie may have received very few or even no ratings. The typically poor performance of machine learning models in this low-data regime is often referred to as the cold-start problem (Schein et al., 2002; Lika et al., 2014; Lam et al., 2008), which is prevalent not only in recommender systems but also in applications where high quality data is sparse. This presents a key challenge: the adaptation of a deep learning model to accurately predict the new feature values in the low data regime. On one hand, it is often required to deploy the model in applications immediately upon the arrival of new features, so it is impractical for the adaptation to wait until much more data has been acquired. On the other hand, simply retraining the model every time a new feature is introduced is computationally costly, and may fall victim to severe over-fitting if there are only a small number of observations available for the new feature.
  Few-shot learning (Snell et al., 2017; Requeima et al., 2019; Vinyals et al., 2016; Gordon et al., 2018) has seen great successes in recent years, particularly in image classification tasks; however, these approaches typically treat all tasks as independent of one another. We wish to extend these ideas to the challenge of extending deep learning models to new output features, using a method which captures how a new feature relates to the existing features in the model. Furthermore, we seek a method that is computationally efficient, ideally requiring no fine-tuning of the model, and that is resistant to over-fitting in the few-shot regime.
  To address these needs simultaneously, our contributions are as follows:
  • We propose an auxiliary neural network, called Contextual HyperNet (CHN), that can be used to initialize the model parameters associated with a new feature (see Section 2). CHNs are conditioned on both a context set made up of observations for the new feature, and any associated content information or metadata. CHNs amortize the process of performing gradient descent on the new parameters by mapping the newly observed data directly into high-performing new parameter values, with no additional fine-tuning of the model being required. This makes CHNs highly computationally efficient and scalable to large datasets. • We use a CHN to augment a partial variational autoencoder (P-VAE) and evaluate the
  system’s performance across a range of applications (see Section 4). While CHNs are applicable to a wide range of deep learning models, in this work we choose a P-VAE as the evaluation framework. The result is a flexible deep learning model able to rapidly adapt to new features, even when the data is sparsely-observed, e.g. in recommender systems. We show that this model outperforms a range of baselines in both predictive accuracy and speed across recommender system, e-learning and healthcare tasks.
sections.heading: 2 MODEL
sections.heading: 2.1 PROBLEM SETTING
sections.text: Our goal is to enable fast adaptation of a machine learning model when new output features are added to augment the originally observed data. Specifically, we consider the original observations as a set of vector-valued data points D = {x(i)}mi=1, where each of the feature values in a given data point may be missing. We denote xj as the jth feature of a data point x ∈ D and group the observed and unobserved features within a data point as x = [xO,xU ]. In many scenarios, such as recommender systems, a machine learning model p(xU |xO) aims then at predicting the unobserved features xU given observed ones xO.
  Now suppose a new output feature xn becomes available, so that each data vector x ∈ D is augmented to become x̃ = [x;xn]. This happens when e.g. a new item is added to a recommender system, or a new type of diagnostic test is added in a medical application. We note that not every data point x ∈ D receives an observed value for the new feature: a newly added movie may have received very few ratings, or a new diagnostic test may have yet to be performed on all of the patients. We refer to the set of data points where the new feature is observed as the context set for the new feature n, i.e.
  Cn = {x̃ = [x;xn] | x ∈ D, xn is observed}. context set is shown in yellow in Figure 1 Its complement, the target set Tn, is the set of those data points for which there is no associated observation for the feature: Tn = {x̃ = [x;xn] | x ∈ D, xn is unobserved}.
  One can also split the augmented data into observed and unobserved parts, i.e. x̃ = [x̃O, x̃U ]. Using this notation, it is clear that x̃O = [xO;xn], x̃U = xU for x̃ ∈ Cn, and x̃O = xO, x̃U = [xU ;xn] for x̃ ∈ Tn. In addition, we may also have access to some metadataMn describing the new feature. This could be categorical data such as the category of a product in a recommender system or the topic of a question in an e-learning system, or some richer data format such as images or text.
  We wish to adapt the machine learning model pθ0(xU |xO) to pθ(x̃U |xO) so that it is able to accurately predict the value of the unobserved new features for data points x̃ ∈ Tn. A naive strategy would ignore the previous model pθ0(xU |xO) and instead seek the maximum likelihood estimates (MLE) of the parameters for the new model pθ(x̃U |xO). This is typically done by training the new model on the context set, by temporarily moving the observed new features xn to the prediction targets:
  θ̂ = argmax θ [∑ x̃∈Cn log pθ (xn,xU |xO) ] .
  However, in deep neural networks, the number of model parameters θ may be extremely large, so that maximising this log-likelihood is very expensive, particularly if new features are being introduced on a regular basis. Furthermore, optimising θ for one particular feature may lead to poor performance for another, as is the case in catastrophic forgetting (Kirkpatrick et al., 2017) in continual learning tasks. In order to address both of these concerns, we divide the model parameters into parameters θ0 inherent from the old model, and feature-specific parameters θn associated solely with the new feature. In other words, we use pθ0(xU |xO) as a base model and pose a factorisation assumption on the augmented model as pθ(x̃U |xO) = pθ0(xU |xO)pθ0(xn|xO;θn), which together yield a predictive model for the new feature. We then hold θ0 fixed and only seek MLEs for θn. While this greatly reduces the dimensionality of the parameter space over which we optimize for a new feature, and decouples the optimization of parameters for one new feature from another, several issues still exist. This factorization still requires a gradient descent procedure, which can be computationally costly and risks severe overfitting when there is little data for the new feature. Furthermore, it is not immediately clear how to make the estimation of θn depend on the feature metadataMn. To address these problems, we introduce a Contextual HyperNetwork (CHN) Hψ(Cn,Mn), an auxilliary neural network that amortizes the process of estimating θn. The goal is that when a new feature xn∗ is added at test time, the CHN will directly generate “good” parameters θ̃n∗ = Hψ(Cn∗ ,Mn∗) such that the new predictive model pθ0(xn∗ |xO;θn∗ = θ̃n∗) can predict the values of the new feature accurately.
sections.heading: 2.2 CONTEXTUAL HYPERNETWORKS
sections.text: CHNs aim to map the context set Cn and metadataMn into an estimate of the new model parameters θ̃n. Since the size of Cn is variable for each feature, CHNs require an architecture that is able to adapt to a varying input dimension. This challenge is addressed through the use of a PointNet-style set encoder (Qi et al., 2017; Zaheer et al., 2017). For each context point x̃(i) ∈ Cn, we concatenate the new feature x(i)n with a fixed-length encoding z(i) (see below) of the other observed features x (i) O within the data point. Each of these concatenated vectors [z (i), x (i) n ] is then input to a shared neural network f(·), and the outputs f([z(i), x(i)n ]) are aggregated by a permutation-invariant function such as summation in order to produce a single, fixed-length vector. This output is passed through a second neural network g(·) to produce a fixed-length encoding of cn we term a “context vector". This architecture is displayed in Figure 2a.
  The fixed length encoding z(i) of the observed features x(i)O for each context data point x̃ (i) ∈ Cn is obtained by inputting the observed features to the base model pθ0(xU |xO) and taking some intermediate representation from within the model: in an autoencoder model, this could be the encoded vector representing the data point at the information bottleneck, while in a feed-forward model it could be the output of an intermediate layer. By encoding the existing features in this way, we hope to enable the CHN to interpret the observed values for the new features in the context of the base model’s representation of remainder of the data point.
  Additionally, any feature metadataMn is passed through a neural network h(·) to produce a fixed length metadata embedding vector mn. In the case of image or text metadata, specialized architectures
  such as convolutional or sequence models can be used here. The concatenated vector [cn;mn] is then input into a final feed-forward neural network which outputs the new feature-specific parameters θ̃n. CHNs can be applied to any predictive model with dynamically added output features: Figure 2b shows the application of CHN to an predictive model. As an example, we illustrate how CHN is used with an autoencoder-style model in Figure 3.
  Since it is possible to parallelize the encoding of the context set Cn both across each data point x(k) and across the observed features within each data point xi, the computational costs for parameter prediction with CHNs are able to scale efficiently with both the context set size and the number of observed values within each data point in the context set. When combined with the lack of any costly iterative gradientdescent procedure, this makes CHNs an extremely efficient choice for parameter initialization.
sections.heading: 2.3 TRAINING CHNS WITH META-LEARNING
sections.text: We adopt a meta-learning approach to training the CHN, treating the prediction of the values of each new feature as an individual task, with the aim of producing a model that can “learn how to learn” from Cn andMn. First, a base model pθ0(xU |xO) is trained on the data observed before the adaptation stages; this model is then frozen during CHN training. To implement the training strategy, in the experiments we divide the dataset into three disjoint sets of features (see Figure 4): a ‘training’ set for base model training in the first stage, a ‘meta-training’ set for CHN meta-learning in the second stage, and a meta-test set for CHN evaluation in the third stage.
  Meta-Training of the CHN During meta-training, the parameters θ0 of the base model are frozen, and we learn the parameters ψ of the CHN. We iterate the following training steps in mini-batches of features B sampled from the meta-training set for every step:
  1. For each feature n in B, sample kn data points in which this feature is observed to form the context set Cn, and reveal the associated feature values to the model. In our experiments we sample kn ∼ Uniform[0, ..., 32] to ensure that a single CHN can perform well across a range of context set sizes.
  2. For each feature n ∈ B, compute feature-specific parameter predictions using the CHN,
  θ̃n = Hψ(Cn,Mn).
  3. For each feature n ∈ B, estimate the log-likelihood of the CHN parameters ψ given the ground truths for the hidden values of the feature n in the data points in its target set Tn, using the augmented model pθ0(xn|xO, θ̃n):
  l(ψ) = 1∑ n∈B |Tn| ∑ n∈B ∑ {i|x(i)∈Tn} log pθ0(x (i) n |x (i) O ; θ̃n).
  4. Update the CHN parameters by taking a gradient ascent step in ψ for l(ψ).
  Note that the log-likelihood is only computed for the hidden values of the new feature in the target set Tn, and not for the observed values in Cn – this is to ensure that the CHN produces parameters which generalize well, rather than overfitting to the context set. This approach is consistent with many meta-learning methods such as MAML (Finn et al., 2017), where the meta-learning model is updated based on its performance on a “test set" of previously unseen examples for each new task.
  Evaluating the CHN At evaluation time, the parameters of both the base model and the CHN are now frozen. A fixed context set and metadata are provided for each feature in the meta-test set, and
  these are used to initialize feature-specific parameters for the meta-test features using the trained CHN. These parameters are then used to make predictions for all of the target set values for the new features, from which evaluation metrics are computed.
  3 RELATED WORK
  CHNs aim to solve the problem of adapting to a new feature with very few available observations, and thus relate to few-shot learning and related fields such as meta-learning and continual learning. From a technical point of view, we use an auxiliary neural network to amortize the learning of parameters associated with the new feature, which falls under the domain of hypernetworks. Furthermore, in the context of recommender systems, a number of related methods have been proposed to address the cold-start problem. We thus discuss related work in these three areas.
  Few-Shot Learning Few-shot learning is the problem of designing machine learning models that can adapt to new prediction tasks given a small number of training examples. A popular approach to this problem is gradientbased meta-learning, such as MAML (Finn et al., 2017) and Reptile (Nichol & Schulman, 2018), which seek a parameter initialisation θ that can rapidly adapt to tasks drawn from a task distribution p(T ).These methods do not directly condition the parameter initialisation for a
  new task on any associated data or metadata, instead relying on fine-tuning,which can be both computationally expensive and lead to severe overfitting when little data is available. Another line of methods seek to adapt a classifier to a task based on a context set of class examples. For instance, by embedding class examples to provide a nearest neighbours classifier (Snell et al., 2017), learning an attention mechanism between class examples and a new example (Vinyals et al., 2016), or modulating activation functions (Munkhdalai et al., 2017) within a feature extractor conditioned on the context set. Conditional Neural Adaptive Processes (Requeima et al., 2019), which are based on Conditional Neural Processes (Garnelo et al., 2018), adapt both classifier and feature extractor parameters based on the context set for a task. Similarly, Gidaris & Komodakis (2018) generate classifier weights for a new image class based on features extracted using the base model. However, in all cases, each task or image class is treated as independent from all others, whereas CHNs explicitly utilize all previously-observed features in the base model when adapting to a new feature.
  A closely related field is continual learning (Kirkpatrick et al., 2017; Nguyen et al., 2017), where a model seeks to adapt to new tasks or a shifting data distribution while avoiding catastrophic forgetting of old tasks. Continual learning does not necessarily address the few-shot scenario and is commonly applied in classification settings where the classifier/heads can be either shared or independent. CHNs can be seen as a means of addressing continual learning in the few-shot learning regime, by generating parameters for a new feature conditioned on all of the features already learned by the model.
  Hypernetworks Hypernetworks (Ha et al., 2016) are auxiliary neural networks which generate the parameters of a neural network. They were introduced in Ha et al. (2016) as a form of model compression, with the hypernetwork taking as in input only structural information about the weight matrix they are currently predicting, such as its index in a sequence of layers. By contrast, CHNs are explicitly conditioned on data relevant to the weights currently being predicted. Bertinetto et al. (2016) train a hypernetwork to predict all of the parameters of a binary classifier for a class of images, conditioned on a single exemplar image for the class. It is found that the output dimension of the hypernetwork grows extremely large for even small classifiers—in order to mitigate this, the authors propose a factorisation of the parameters, whereas we instead choose to learn only a small number of feature-specific parameters θn. Task-conditioned hypernetworks (von Oswald et al., 2019) provide an application of hypernetworks to multi-task continual learning, where weights for the entire neural network for different tasks are predicted using a hypernetwork, based on a learned task embedding.
  This setting differs from our work as the continual learning tasks are assumed to be independent, and the hypernetwork is not conditioned directly on data for the new task, instead requiring a gradient descent process to learn the associated task embedding.
  Cold Starts in Recommender Systems Cold starts (Schein et al., 2002; Lika et al., 2014) occur when there is little or no data associated with a novel item or user in a recommender system. Collaborative filtering approaches to recommender systems have enjoyed great success for many years (Schafer et al., 2007; Sarwar et al., 2001; He et al., 2017), but can fail completely when there is very limited rating data associated with a new user or item (Lam et al., 2008). One potential solution to cold starts is given by content-based methods (Pazzani & Billsus, 2007; Lops et al., 2011), which use any available descriptive information about the new user or item. Hybrid approaches (Balabanović & Shoham, 1997; Stern et al., 2009; Gomez-Uribe & Hunt, 2015) seek to marry these two approaches, making use of both collaborative and content-based methods. Meta-learning approaches also show promise for solving cold starts, including MAML-like approaches (Bharadhwaj, 2019) for initialising new items. Vartak et al. (2017) takes a similar approach to CHNs by adapting the weights of a classifier, but the adaption is based on user rather than item history, and it relies on class-specific item embeddings and thus is not extensible to regression settings. In this work, when applied to recommender systems, CHNs combine the strengths of all of these approaches, using content information, ratings data and latent representations of the associated users to generate accurate parameters for novel items. Recently, graph neural network (GNN)-based approaches (You et al., 2020; Schlichtkrull et al., 2018; Wang et al., 2019; Hamilton et al., 2017) have shown promise when applied to recommender systems. These methods formulate each item and each user as a node, and incorporate new items by adding nodes to the graph, where the graph structure may be exploited for inductive reasoning about new features. In this work, we target more standard, vector-based neural networks which are currently deployed across a wide range of application domains where GNNs may not be suitable, but future work may explore the application of CHNs to GNNs and the relationships between the two.
sections.heading: 4 EXPERIMENTS
sections.text: We demonstrate the performance of the proposed CHN in three different real-world application scenarios, including recommender systems (Section 4.2), healthcare (Section 4.3) and e-learning (Section 4.4). Our method exhibits superior performance in terms of prediction accuracy across all these applications. We also demonstrate an advantage in terms of computational efficiency in a large scale real-life setting.
sections.heading: 4.1 EXPERIMENT SETTINGS
sections.text: In all our experiments, we apply a CHN to a partial variational autoencoder (P-VAE) (Ma et al., 2018b;a) as an exemplar model. This is a flexible autoencoder model that is able to accurately work with and impute missing values in data points, allowing us to model sparsely-observed data such as that found in recommender systems. For each new feature n, we augment the P-VAE’s decoder with a new decoder head consisting of an additional column of decoder weights wn and an additional decoder bias term bn which extend the model’s output to the new feature, so that θn = {wn, bn}. Figure 3 illustrates how a CHN is used to extend a P-VAE to make predictions for a new feature xn.
  For all experiments, we train the CHN to output accurate feature parameters based on a range of context set sizes k ∈ [0, ..., 32] by randomly sampling k on each occurrence of a meta-training set feature. We then evaluate the performance of the CHN and baselines on the meta-test set features for a fixed range of context set sizes, ensuring that the same context sets are revealed to the CHN and each baseline. Further experimental results can be found in Appendix C, and full details on the model architectures and hyperparameters can be found in Appendix D. All results are averaged across 5 random train/meta-train/meta-test feature splits, and ±1σ error bars are plotted across these splits. Hyperparameters and model architectures were tuned on different data splits to those used in the final experiments.
  We consider the following baselines for generating the new feature parameters θn = {wn, bn}. All methods are applied to the same base trained P-VAE model to ensure a fair comparison.
  • Random: Generate a random value for θn for each new decoder head using Xavier initialisation for weights and 0 for biases. • Mean Imputing: Set weights and biases to always predict the mean of the observed values for the new feature in the context set, i.e. wn = 0 and b = σ−1 ( 1 k ∑ i∈Cn x (i) n ) . • Mean Head Parameters: Generate the new head parameters θn as the mean of all of the head parameters learned on the training set features. • Mean Head Parameters (Matching Metadata): As above, but averaging only over parameters of heads whose associated feature has metadata categories matching those of the new feature. • k-Nearest Neighbour Head Parameters: Generate the new head parameters θn as the mean of the head parameters of the k-nearest neighbour features in terms of Euclidean distance, where column-wise mean imputing is used to fill in unobserved values. • Train from Random: Initialize the new feature head parameters randomly, and then fine-tune these parameters on the data in the context set Cn for a fixed number of epochs. • MAML: We meta-learn an initialisation of θn using Model-Agnostic Meta Learning (Finn et al., 2017), where we treat the prediction of each feature as a separate task and fine-tune these parameters on the context set. In all experiments, we compare with the MAML baseline which has the bestperforming number of fine-tuning epochs. For full details, see Appendix B.
sections.heading: 4.2 RECOMMENDER SYSTEMS
sections.text: In real-life recommender systems, new users and new items are continuously added as new customers join and new products are launched. In deep learning based frameworks such as Sedhain et al. (2015); Liang et al. (2018); Gong et al. (2019); Ma et al. (2018a), the deep neural networks are commonly used in a user-based manner. In this approach, each new user is treated as a new data point, while each new item is treated as a new feature. To add a new item, one must extend the network architecture to incorporate the new feature, and we propose CHNs as an efficient way to predict the parameters associated with the new feature.
  We evaluate the scenario above with MovieLens-1M dataset (Harper & Konstan, 2015). The dataset consists of 1 million ratings in the range 1 to 5 given by 8094 users to 5660 movies, and is thus 2.2% observed. For each movie, we have associated metadataMn giving a list of genres associated with the movie, such as Action or Comedy, which we encode in a binary format, alongside the year of release which we normalize to lie within [0, 1]. For each random data split, we sampled 60% of movies as training data to train the base P-VAE model, used 30% as a meta-training set for CHN training and used the remaining 10% as a meta-test set.
  The plot in Figure 5 (left) shows the performance of our proposed CHN, comparing with all other baselines in terms of RMSE (lower is better). Our method shows an advantage over all considered
  baselines other than MAML in the few-shot regime (k < 8), while achieving competitive performance with MAML without requiring costly fine-tuning.
sections.heading: 4.3 HEALTHCARE
sections.text: In healthcare applications, a new question is often added to an existing health-assessment questionnaire, and in hospitals, new medical devices may be introduced to make physiological measurements. In this case it is desired for a model to quickly adapt to the newly added feature for health assessment, even when relatively few tests have been administered and so data is scarce.
  We assess the utility of CHNs in a healthcare setting using synthetic data generated by the Neuropathic Pain Diagnosis Simulator (Tu et al., 2019). This simulator produces synthetic data using a generative model to simulate pathophysiologies, patterns and symptoms associated with different types of neuropathic pain. The data is binary, where a 0 represents the a diagnostic label that is not present in a patient’s record, and a 1 indicates a diagnostic label that is present. We simulated 1000 synthetic patients, and removed features with fewer than 50 positive diagnoses, resulting in 82 remaining features, with 17.3% of the values in the dataset being positive diagnoses. We used 50% of the features as training set; 30% of the features as the meta-test set and 20% of the features as the meta-test set.
  The plot in Figure 5 (middle) shows the results in terms of AUROC (higher is better), as the dataset is highly imbalanced. Our method consistently outperforms all baselines across all values of k, while many methods including MAML suffer from severe overfitting when k is small. In contrast to the MovieLens-1M result, here the 10-nearest neighbour approach does not seem to leverage more datapoints in the context set. This shows that our method is desirable in the cost-sensitive healthcare environment, even for highly imbalanced medical tests where results are largely negative.
sections.heading: 4.4 E-LEARNING
sections.text: We foresee CHNs being valuable in online education settings, potentially allowing teachers to quickly assess the diagnostic power of a new question given a small number of answers, or to gauge whether a new question’s difficulty is appropriate for a particular student.
  We assess the performance of the CHN in an e-learning setting using a real-life dataset provided by the e-learning provider Eedi for the NeurIPS 2020 Education Challenge (Wang et al., 2020). In particular, we use the dataset for the first 2 tasks, filtered so that all students and questions have at least 250 associated responses. This results in a dataset of for 6797 students across 4792 questions, detailing whether or not a student answered a particular question correctly. The dataset contains approximately 2.7 million responses, making it 8.2% observed. We treat each student as a data point and each question as a feature, and use a binary encoding of each question’s associated subjects as metadata. We used 60% of the questions as training set; 30% of the questions as the meta-test set and 10% of the questions as the meta-test set.
  The right panel in Figure 5 illustrates the performance on prediction on the unseen meta-test set in terms of AUROC. Our method shows a significant improvement over all of the considered baselines over the entire range of k, suggesting promise for applying CHNs in educational settings.
  4.5 TIMING EXPERIMENTS
  We used the E-learning dataset to compare the time taken to generate new feature parameters at meta-test time for a number of methods considered in our experiments. The results are shown in Table 1. We see that the CHN offers nearly a 4-fold speedup compared to the nearestneighbours based approach. We see a similar difference in performance when compared to training the new heads on a single observation for just 10 epochs. Moreover,
  while this training time grows rapidly with the number of observations in the context set, the time taken for a CHN remains nearly constant since it can efficiently parallelize along these observations, making CHNs an extremely efficient initialisation choice for larger context set sizes. The experimental settings are specified in Appendix D where the total computation time for training and evaluating a CHN was approximately 3 minutes on the Neuropathic Pain dataset, 1.5 hours on the E-learning dataset, and 8 hours on MovieLens-1M.
sections.heading: 5 CONCLUSION
sections.text: We introduce Contextual HyperNetworks (CHNs), providing an efficient way to initialize parameters for a new feature in a model given a context set of points containing the new feature and feature metadata. Our experiments demonstrate that CHNs outperform a range of baselines in terms of predictive performance across a range of datasets, in both regression and classification settings, and are able to perform well across a range of context set sizes, while remaining computationally efficient. In the future work, we will evaluate CHNs in streaming setting with large-scale real-world applications.
sections.heading: A APPENDIX: PARTIAL VARIATIONAL AUTOENCODERS
sections.heading: A.1 PARTIAL VARIATIONAL AUTOENCODERS
sections.text: For our experiments, we base our model on the Partial Variational Autoencoder (P-VAE) (Ma et al., 2018a) - this model combines a traditional variational autoencoder (VAE) model with a PointNet-style set encoder (Qi et al., 2017), allowing it to efficiently encode and reconstruct partially observed data points. The P-VAE is based on the observation that typically the features in a VAE are assumed to be conditionally independent when conditioned on the latent variable z. That is,
  p(x|z) = ∏ j p(xj |z)
  Then, given a data point x with observed features xO and unobserved features xU , we have that
  p(xU |xO, z) = p(xU |z)
  Hence, if we can infer a posterior distribution over z from the observed features, we can use this to estimate p(xU |xO). The P-VAE infers a variational posterior distribution over z using an amortized inference network (or encoder network) qθ(z|xO) and approximates the conditional data distribution given a value of z using a decoder network pφ(xO,xU |z). In our model, we extend the decoder to decode the value of a new feature xn by initialising an additional subnetwork in the decoder which we term a decoder head, with parameters φn, to extend its output dimension by one. In principal this head could be of any architecture which takes as input the output of the shared layers of the decoder, but in practice we found that simply extending the final layer of weights and biases to accommodate a new output dimension yielded good results while remaining parameter-efficient as the number of output features grows.
sections.heading: A.2 TRAINING P-VAES
sections.text: The P-VAE is trained to reconstruct observed features in the partially-observed data point, and in the process learn to infer a variational posterior qθ(z|xO) over the latent variable z. The P-VAE is given batches of data points where features from both the meta-train and meta-test sets are hidden from the model. Additionally, each time a particular data point is input, some additional features are also randomly hidden from the model using a Bernoulli mask, in order to ensure the model is robust to different sparsity patterns in the data. The P-VAE is then trained by maximising the Evidence Lower-Bound (ELBO) (Ma et al., 2018b):
  log p(xO) ≥ log p(xO)−DKL(q(z|xO)||p(z|xO)) = Ez∼q(z|xO) [log p(xO,xU |z)]−DKL [q(z|xO)||p(z)] = Lpartial(xO)
sections.heading: B APPENDIX: BASELINES
sections.text: Here we provide additional details and results for the baselines used in our experiments.
sections.heading: B.1 MAML
sections.text: We adapt the Model-Agnostic Meta Learning (Finn et al., 2017) technique as a baseline. The decoder head parameters θn are adapted using the MAML algorithm in the ‘meta-training’ stage. Each new feature X is viewed as a separate MAML task, with some observed and unobserved values. We sample the tasks in batches of size M and train the inner (a.k.a. fast) model over N steps. The inner model training loss is the ELBO of the PVAE on the observations LXO . The meta-model (a.k.a. the slow or outer model) is trained by being given the context set observations, and computing a reconstruction loss on the target set, L̂T ,C(fθ′ ,X ). The gradient for the meta-model update is taken over the batch reconstruction losses mean. The full algorithm is detailed in Algorithm 1.
  Algorithm 1 Feature-wise Model-Agnostic Meta-Learning with PVAE Input: p(X ): distribution over features. α, β: learning rate hyperparameters. M : meta-batch size, N : number inner iterations.
  Initialize θ while not done do
  Sample M features Xi ∼ p(X ). for all Xi do θi,0 ← θ for j ← 0, N do
  Evaluate ELBO gradient ∇θi,jLXO (fθi,j ,Xi) w.r.t. observations in K examples Optimize inner model parameters: θi,j+1 ← θi,j −∇θi,jLXO (fθi,j ,Xi)
  end for θ′i ← θi,N
  end for Evaluate gradient of mean reconstruction error∇θ 1M ∑ Xi∼p(X ) L̂Ti,Ci(fθ′i ,Xi)
  Optimize meta-model parameters: θ ← θ −∇θ 1M ∑ Xi∼p(X ) L̂Ti,Ci(fθ′i ,Xi)
  end while Output: θ
  Notably, since MAML aims to fit parameters that adapt quickly to new tasks, it allows for fine-tuning at evaluation time, that is, training the model for several iterations from the MAML parameter initialization. Here, we evaluate the model with and without fine-tuning.
  In the MAML baseline experiments we use M = 4, N = 10, ADAM (Kingma & Ba, 2014) with learning rate α = β = 10−2 for inner and outer model optimization. The model fine-tuned performance is evaluated over {1, 3, 5, 10} epochs and beset results are used. We make use of the higher order optimization facilitated by the higher library (Grefenstette et al., 2019) in the implementation of this baseline.
sections.heading: C APPENDIX: ADDITIONAL RESULTS
sections.text: Here we provide additional results not included in the main text.
sections.heading: C.1 MAML
sections.text: Figure 6 shows the performance of the MAML baseline for different numbers of fine-tuning epochs and with no fine-tuning. As expected, the baseline with no fine-tuning is outperformed by those where fine-tuning is employed. For the Neuropathic Pain and E-learning datasets, the increase in the number of fine-tuning epochs corresponds to improvement in performance (greater AUROC), whereas in case of MovieLens-1M, performance drops (RMSE increases) with longer fine-tuning, particularly for the smaller context set sizes.
sections.heading: C.2 K-NEAREST NEIGHBOUR HEAD PARAMETERS
sections.text: We consider k-Nearest Neighbour Head Parameters baselines for the values k ∈ {1, 5, 10}. Figure 7 shows the performance of this baseline for the different values of k across a range of context set sizes. We expect that as k is increased further, and the number of head parameters averaged over grows, the behaviour will approach that of the mean head parameter baselines. In the main text, 10-Nearest Neighbours is used throughout, as it yields good performance in both the low and high-data regimes.
sections.heading: C.3 FINE-TUNING
sections.text: In the main text, we show the performance of training the new decoder heads on their context sets from randomly initialized parameters for 10 epochs (see results for “Train from Random (10 epochs)”), in order to provide a trade-off between predictive accuracy and computational cost. This section aims to elaborate further on this trade-off. In Figure 8a, we show the predictive performance of the P-VAE on the meta-test set after training randomly initialized head parameters for an increasing number of epochs, for a range of context set sizes k. We see that the performance improves with training in all cases, with better performance achieved as the context set size k increases, and thus the effect of over-fitting is lessened.
  Furthermore, in Figure 8b, we perform the same experiment but instead initialising the heads with the CHN parameters. We see that in all cases except k = 0 and k = 32, training by gradient descent leads to a decrease in performance due to over-fitting, suggesting that the CHN has an implicit regularising effect on the parameter initialisation. We note also that in all cases, the untrained CHN parameters substantially outperform those trained from the random initialisation for all values of k, even after 25 training epochs, with many of the training curves appearing to approach convergence.
sections.heading: C.4 RMSE VALUES FOR BINARY DATASETS
sections.text: In Figure 9 we provide the RMSE values (lower is better) for the main paper experiments run on the Neuropathic Pain and E-learning datasets. We see that the relative performance ordering of methods in terms of RMSE is largely identical to the performance ordering obtained from the AUC results, suggesting that the proposed CHN approach is indeed better than other baselines.
sections.heading: C.5 METADATA ABLATION STUDY
sections.text: It is possible to obtain an ablation for removing the usage of the context set Cn as an input to the CHN by looking at the performance for k = 0 in any of the preceding performance plots. In order to further investigate the effect of metadata on the CHN’s performance, we additionally perform an ablation on MovieLens-1M by removing the use of metadata (movie genre and year) as an input to the CHN. We also compare the performance of averaging all trained head parameters, vs. averaging all trained head parameters whose features are a subset of those of the new feature under consideration. The results of these ablations can be found in Figure 10. We see that in both cases, in absolute terms the difference in performance due to the metadata is slight, with a small improvement made to the mean head parameter performance, and a very slight improvement to the CHN performance when k = 0. This is likely due to the metadata carrying very little information for this dataset relative to the data points themselves, so that when k ≥ 1 the CHN is able to effectively disregard the metadata in favour of the observed data points. Future work could investigate the performance of CHNs on datasets with richer and more information feature metadata.
sections.heading: D APPENDIX: EXPERIMENT DETAILS
sections.text: All models were implemented in PyTorch (Paszke et al., 2017). All experiments were performed on a single Nvidia Tesla K80 GPU. For training both the P-VAE and the CHN’s parameters, the ADAM (Kingma & Ba, 2014) optimizer was used with β1 = 0.9, β2 = 0.999 and ε = 10−8
  Details of hyperparameters and model architectures used for each dataset can be found in Table 2.