A recent approach to few-shot classification called matching networks has demonstrated the benefits of coupling metric learning with a training procedure that mimics test. This approach relies on an attention scheme that forms a distribution over all points in the support set, scaling poorly with its size. We propose a more streamlined approach, prototypical networks, that learns a metric space in which few-shot classification can be performed by computing Euclidean distances to prototype representations of each class, rather than individual points. Our method is competitive with state-of-the-art few-shot classification approaches while being much simpler and more scalable with the size of the support set. We empirically demonstrate the performance of our approach on the Omniglot and miniImageNet datasets. We further demonstrate that a similar idea can be used for zero-shot learning, where each class is described by a set of attributes, and achieve state-ofthe-art results on the Caltech UCSD bird dataset.
1 INTRODUCTION
One-shot classification (Miller et al., 2000; Lake et al., 2011; Koch, 2015) (and more generally, fewshot classification) is a problem in which a classifier must be adapted to accommodate new classes not seen in training, given only a single (n) example(s) of these classes. A classical approach, such as retraining the model on the new data, would severely overfit. While the problem is quite difficult, it has been demonstrated that people have the ability to successfully perform one-shot classification (Lake et al., 2011). Nonparametric models such as nearest neighbors are useful in one-shot classification because they naturally adapt to new data, however this comes at the cost of storing the entire set of examples per class, the “support set”.
To overcome this, much progress has been made recently in applying metric learning (Goldberger et al., 2004; Kulis, 2012; Bellet et al., 2013) to one-shot tasks. Most recently, (Vinyals et al., 2016) proposed a metric learning approach that they call matching networks. This approach uses an attention mechanism over a learned embedding of the support set in order to predict class labels for the points to be classified, a.k.a the “query set”. It optionally allows the embeddings to be conditioned on other points in the support set (“full context embeddings”) or for the embeddings to be fine-tuned at test time. A particularly interesting feature of the matching networks model is that it utilizes sampled mini-batches called “episodes” during training, where each episode is designed to mimic the one-shot task. This makes the training problem more faithful to the test environment. Matching networks however optionally utilize additional components such as an attention-based LSTM to change the embedding based on the support set. This complexity makes implementation more difficult in addition to the aforementioned poor scaling characteristics due to computing attention over the entire support set.
In this paper, we propose a few-shot learning classifier based on a relatively simple idea: there exists an embedding, where points belonging to a class cluster around a single prototype. This inductive bias is a useful one to combat overfitting for one-shot tasks. Our approach also comes with the benefit that it is very simple to implement, and computationally fast. In order to do this, we learn a non-linear mapping of the input into an embedding space using a neural network, and take the class
∗Most work done while author was at Twitter.
prototype to be the mean of the support set in the embedding space. Classification is then performed by simply finding the nearest prototype to the embedded query point. We find that this approach yields competitive results with matching networks and other one-shot learning approaches, despite being much simpler.
A related problem is known as zero-shot learning, where instead of being given a small number of examples of a new class at test-time, each class comes with a set of meta-information, often attributes, that give a high level description of that class. The idea then is to learn a mapping from input examples to the high-level attributes of their member class. We adapt the idea of prototypical networks to this setting by learning a secondary embedding of the attribute vector such that the image embeddings and attribute embeddings lie within the same space. In this case, we use the attribute embedding as the class prototype, rather than the class mean.
2 RELATED WORK
Neighborhood Components Analysis (NCA) (Goldberger et al., 2004) learns a Mahalanobis distance to maximize K-nearest-neighbour’s (KNN) leave-one-out accuracy in the transformed space. A distribution over the neighbors of each data point is computed according to a softmax over the corresponding Mahalanobis distances. This distribution is marginalized to form a distribution over class assignments and the projection matrix is updated via gradient descent to maximize the probability of the true class. (Salakhutdinov & Hinton, 2007) extend NCA by using a neural network to perform the transformation. Our approach is similar in that we optimize a softmax based on distances in the transformed space. Ours differs because it is a softmax over classes, rather than points, computed from Euclidean distances to each class’s prototype representation. This is more appropriate for few-shot learning for two reasons: (a) the number of support points can vary by class, and (b) each class has a succinct representation independent of the number of data points, and this representation can optionally be updated in an online manner.
Our approach is similar to the nearest class mean approach of (Mensink et al., 2013) from the metric learning literature, where each class is represented by the mean of its examples, and classification is performed by finding the prototype that is closest to the query point. Their approach was developed to rapidly incorporate new classes into a classifier without retraining, however it relies on a linear embedding and is designed to handle the case where the novel classes come with many examples. In our approach, we utilize neural networks to learn a non-linear embedding of the features and we couple this with episodic training in order to handle the one-shot scenario. Mensink et al. do attempt to extend their approach to perform non-linear classification, but they do this by allowing classes to have multiple prototypes. They find these prototypes in a pre-processing step by using k-means on the input space, and then perform a multi-modal variant of their linear embedding. By contrast, we learn a non-linear embedding in an end-to-end manner with no such pre-processing, producing a non-linear classifier that still only requires one prototype per class.
In matching networks (Vinyals et al., 2016) they propose a meta-learning strategy in which training mimics test by stochastically creating one-shot “episodes”. We adopt the same strategy when training our models. They, like us, use neural networks to non-linearly transform data points into a space that is more amenable to classification. However, matching networks make predictions by computing attention weights over each point in the support set. This becomes computationally expensive as the size of the support set grows. Our approach, on the other hand, first summarizes each class in the support set by a prototype and then computes a distribution over classes. Ours thus has flexibility in the way the prototypes are computed and can handle additional support points gracefully by updating prototypes online.
The neural statistician (Edwards & Storkey, 2016) extends the variational autoencoder (Kingma & Welling, 2013) to learn generative models of datasets rather than individual points. One component of the neural statistician is the “statistic network” which summarizes a set of data points into a statistic vector. It does this by encoding each point within a dataset, taking a sample mean, and applying a post-processing network to obtain an approximate posterior over the statistic vector. Edwards & Storkey test their model for one-shot classification on the Omniglot dataset (Lake et al., 2011) by considering each character to be a separate dataset and making predictions based on the class whose approximate posterior over the statistic vector had minimal KL-divergence from the test point. Like the neural statistician, we also produce a summary statistic for each class. However, ours
is a discriminative model which is more appropriate because our primary task, one-shot learning, is also discriminative. Discriminative training has the added benefit of lending our model more flexibility in both the way we compute summary statistics and use them to make predictions at test time.
There are many other approaches to one-shot learning that employ very different techniques from ours. Koch uses siamese networks to predict the probability that two images belong to the same class. Lake et al. devise a hierarchical Bayesian generative model of how a handwritten character is created in order to perform one-shot learning on the Omniglot dataset. Santoro et al. propose memory augmented neural networks (MANN) that reference an external memory in a similar fashion to neural Turing machines (Graves et al., 2014). This allows them to store support examples in an external memory and reference them later when making classification decisions. They also introduce a form of episodic training, similar to that in matching networks.
3 PROTOTYPICAL NETWORKS
At prediction time we are given a support set ofN labeled examples: S = {(xi, yi)}Ni=1 = S1∪. . .∪ SK where Sk = {(x, y) ∈ S | y = k}. Our method computes a class representation ck, or prototype, of each class through an embedding function fθ(x) parameterized by learnable parameters θ:
ck = 1 |Sk| ∑
(x,y)∈Sk fθ(x) (1)
Given a test point x̃, prototypical networks forms a distribution over classes based on a softmax over the Euclidean distances between its embedding and the prototypes:
p(y = k | x̃) = exp(−‖fθ(x̃)− ck‖ 2)∑
k′ exp(−‖fθ(x̃)− c′k‖2) (2)
Learning proceeds by maximizing the log-probability of the true class ỹ:
max θ
log p(ỹ |x̃) (3)
We train in an episodic manner similar to Vinyals et al. (2016) by randomly selecting a subset of classes from the training set, then choosing a subset of examples within each class to act as the support set and the remainder to serve as test points.
3.1 PROTOTYPE NORMALIZATION
In episodic training, the support set is randomly chosen from among the training points. In datasets with high variability this can lead to a large variance in the class prototypes, c, between episodes. In order to reduce this variability, we found that it can sometimes be beneficial to normalize the prototypes, c̄k = ck‖ck‖ and use c̄k in place of ck in Equation (2). This ensures that the prototypes always lie on the unit sphere, although the query points are still allowed to be embedded off of the unit sphere. Normalization has two benefits: the reduction in variance helps to greatly speed up training, while the restriction of the prototypes to the unit sphere confers additional regularization.
3.2 PREDICTING THE WEIGHTS OF A LINEAR CLASSIFIER
A simple analysis is useful in gaining insight into the nature of the learned classifier (a similar analysis appears in Mensink et al. (2013)). When we use Euclidean distance to measure the distance between a query point and the class prototypes, then the loss function in (2) is equivalent to a linear classifier with a particular parameterization. To see this, we expand the term within the exponent:
−‖fθ(x̃)− ck‖2 = −(fθ(x̃)− ck)>(fθ(x̃)− ck) = −fθ(x̃)>fθ(x̃) + 2c>k fθ(x̃)> − c>k ck (4)
The first term in Equation (4) is constant with respect to the class k, so it does not affect the softmax probabilities. We can write the remaining terms as a linear classifier as follows:
2c>k fθ(x̃)− c>k ck = w>k fθ(x̃) + bk wk = 2ck
bk = −c>k ck
We can view this through the lens of meta-learning, where the model is predicting the weights and biases of a linear classifier using a simple function of the mean of the embedded support set. By contrast, the predictive function in matching networks is a generalization of a nearest neighbor classifier, rather than a linear classifier.
When using prototype normalization, the biases bk will all be 1, and the class weights wk will be restricted to have a norm of 2. In this case, using Euclidean distance becomes proportional to cosine distance.
A natural question is whether it makes sense to use multiple prototypes per class instead of just one. If each support point were to be considered a prototype, then this would be analogous to doing nearest neighbor classification in the embedding space, which would be computationally expensive. On the other hand, if the number of prototypes per class is fixed, then this would require a partitioning scheme. This has been proposed in Mensink et al. (2013) and Rippel et al. (2016), however both methods require a separate partitioning phase that is decoupled from the weight updates, while our approach is simple to learn with ordinary gradient methods. Finally, the equivalence to a linear classifier suggests that this may be sufficient, as all of the required non-linearity can be learned within the embedding function. Indeed, this is the approach that state-of-the-art neural network classification systems currently use, e.g., (Krizhevsky et al., 2012).
3.3 DESIGN CHOICES
There are still a number of design choices that need to be made with this model in order to achieve optimal performance. One such choice is in deciding how many classes we would like the classifier to operate over during each training episode. For example, at test time we might be evaluating on 5-way classification, but at training time we could train each episode with 20-way classification. We found in general that training on a larger number of classes per episode improves performance, even if the number of classes we need to decide between at test-time is fewer.
Another choice involves the possible decoupling of the n in n-shot between training and testing. We could train on 1-shot, but test on 5-shot or vice-versa. We found that it is typically better to match the shot at training and testing; that is, when it comes to the shot, to match the training procedure to the test procedure. We demonstrate this empirically in the Experiments section below.
Finally, we need to specify whether to use prototype normalization. We found that normalization generally acts as a regularizer and speeds up training.
4 EXPERIMENTS
For few-shot learning, we performed experiments on Omniglot (Lake et al., 2011), the miniImagenet version of ILSVRC-2012 (Russakovsky et al., 2015) proposed by (Vinyals et al., 2016). For zeroshot learning, we perform experiments on the 2011 version of the Caltech UCSD bird dataset (CUB200 2011) (Welinder et al., 2010).
4.1 OMNIGLOT
Omniglot (Lake et al., 2011) is a dataset of 1623 handwritten characters collected from 50 alphabets. There are 20 examples associated with each character, where each example was drawn by a different human subject. We follow the procedure of (Vinyals et al., 2016) by augmenting the characters with rotations in multiples of 90 degrees and using 1200 characters for training and the remainder for evaluation. Our embedding architecture mirrors that of Matching Nets and is composed of four blocks of a 64 filter 3 × 3 convolution, batch normalization (Ioffe & Szegedy, 2015), a ReLU nonlinearity and a 2 × 2 max-pooling, resulting in a 64-dimensional output space. The results of our model trained to perform Omniglot classification are shown in Table 1.
We trained prototypical networks using episodes designed for 1-shot learning, i.e., the support sets during training consist of a single input example, and we train using 20-way classification. Our results are as good or better than those reported in matching networks, and to our knowledge represent the state-of-the-art on this dataset using these splits.
4.2 miniIMAGENET
The miniImageNet dataset (Vinyals et al., 2016) is derived from the larger ImageNet dataset (Deng et al., 2009). It consists of 60,000 color images of size 84 X 84 divided into 100 classes with 600 examples each. It is designed for testing one-shot learning algorithms, where 80 classes are chosen for training, and 20 for testing.
Classification results for miniImageNet are shown in Table 2. The embedding architecture we used for miniImagenet is the same as our experiments for Omniglot, though here it results in a 1600- dimensional output space due to the increased size of the images. We trained two versions of prototypical networks, one with episodes containing a single support examples per class (denoted by 1-shot) and one with five support examples per class (denoted by 5-shot). All episodes contained 20 randomly sampled classes, as 20-way classification represents a more difficult task than 5-way. We evaluated both models on 1-shot and 5-shot for 5-way and 20-way classification at test and find that each model performs best on the number of support examples it was trained for.
4.3 CUB ZERO-SHOT CLASSIFICATION
In order to assess the suitability of our approach for zero-shot learning, we also run experiments on the Caltech-UCSD Birds (CUB) 200-2011 dataset (Welinder et al., 2010). In the zero-shot setting, the goal is to classify query images in the absence of any support examples. Instead, class metadata (such as attributes or a textual description) is provided for each of the test classes. We adapt our
few-shot approach to the zero-shot setting by learning to jointly embed images and metadata in a shared space. The embedded metadata serve as class prototypes and classification is performed by embedding the query image and selecting the class whose prototype is nearest in the Euclidean space.
The CUB dataset contains 11,788 images of 200 bird species. We closely follow the procedure of Reed et al. (2016) in preparing the data. We use their splits to divide the classes into disjoint sets of 100 training, 50 validation, and 50 test. For images we use 1,024-dimensional features extracted by applying GoogLeNet (Szegedy et al., 2015) to middle, upper left, upper right, lower left, and lower right crops of the original and horizontally-flipped image1. At test time we use only the middle crop of the original image. For class metadata we use the 312-dimensional continuous attribute vectors provided with the CUB dataset. These attributes encode various characteristics of the bird species such as their color, shape, and feather patterns.
We learned a simple linear mapping on top of both the 1,024-dimensional image features and the 312-dimensional attribute vectors to produce a 1,024-dimensional output space. We apply prototype normalization to the embedded attributes so that the class prototypes are always of unit length. This serves as a form of regularization to help our embedding functions generalize better. The model parameters were optimized according to our objective via SGD with Adam (Kingma & Ba, 2014) at learning rate of 10−4 and weight decay of 10−5. Early stopping on validation loss was used to determine the optimal number of epochs for retraining on the training + validation set.
Table 3 shows that of methods utilizing attributes as class metadata, we achieve state-of-the-art results by a large margin. Our approach is much simpler than that of other recent approaches (Liao et al., 2016) which train an SVM on a learned feature space obtained by fine-tuning AlexNet (Krizhevsky et al., 2012). These zero-shot classification results demonstrate that our approach is general enough to be applied even when the data points (images) are from a different domain relative to the classes (attributes).
Figure 1 shows a t-SNE (Maaten & Hinton, 2008) visualization of attribute embeddings learned using prototypical networks for zero-shot classification. We can see that the embeddings group the bird species by characteristics such as their color and shape.
5 CONCLUSION
We have proposed a simple method called prototypical networks for few-shot learning based on the idea that we can represent each class by the mean of its examples in a representation space learned by a neural network. We train these networks to specifically perform well in the few-shot setting by using episodic training. Prototypical networks are simple to implement, and computationally efficient. We showed that this approach is equivalent to predicting the weights of a linear classifier, where the weights and biases are a function of the prototypes. Prototypical networks achieve stateof-the-art results on the Omniglot dataset, and competitive results on the miniImagenet dataset. We further showed how this approach can be adapted to the zero-shot setting by taking an embedding of an attribute vector for each class to be the prototype. This approach achieves state-of-the-art results on zero-shot classification of the Caltech UCSD birds dataset.
1https://github.com/reedscot/cvpr2016.
ACKNOWLEDGMENTS
We would like to thank Sachin Ravi and Hugo Larochelle for help in setting up the Omniglot and miniImage data. We would also like to thank Renjie Liao for assistance with the CUB-200 zero-shot procedure and Oriol Vinyals for confirming details regarding the Omniglot and miniImagenet splits and matching nets architectures.

*** Paper Summary ***
This paper simplify matching network by considering only a single prototype per class which is obtained as the average of the embedding of the training class samples. Empirical comparisons with matching networks are reported.
*** Review ***
The paper reads well and clearly motivate the work. This work of learning metric learning propose to simplify an earlier work (matching network) which is a great objective. However, I am not sure it achieve better results than matching networks. The space of learning embeddings to optimize nearest neighbor classification has been explored before, but the idea of averaging the propotypes is interesting (as a non-linear extension of Mensink et al 2013). I would suggest to improve the discussion of related work and to consolidate the results section to help distinguish between the methods you outperform and the one you do not.
The related work section can be extended to include work on learning distance metric to optimize a nearest neighbor classification, see Weinberger et al, 2005 and subsequent work. Extensions to perform the same task with neural networks can be found in Min et al, 09 that purse a goal very close to yours. Regarding approaches pursuing similar goals with a different learning objective, you cite siamese network with pairwise supervision. The learning to rank (for websearch) litterature with triplet supervision or global ranking losses is also highly relevant, ie. one example "the query" defines the class and the embedding space need to be such that positive/relevant document are closer to the query than the others. I would suggest to start with Chris Burges 2010 tutorial. One learning class
I am not sure the reported results correctly reflect the state of the art for all tasks. The results are positive on Omniglot but I feel that you should also report the better results of matching networks on miniImageNet with fine tuning and full contextual embeddings. It can be considered misleading not to report it. On Cub 200, I thought that the state-of-the-art was 50.1%, when using features from GoogLeNet (Akata et al 2015), could you comment on this?
Overall, paper could greatly be improved, both in the discussion of related work and with a less partial reporting of prior empirical results.
*** References ***
Large Margin Nearest Neighbors. Weinberger et al, 2005
From RankNet to LambdaRank to LambdaMART: An Overview, Chris J.C. Burges, June 23, 2010
A Deep Non-linear Feature Mapping for Large-Margin kNN Classification, Min et al, 09


Algorithm design is a laborious process and often requires many iterations of ideation and validation. In this paper, we explore automating algorithm design and present a method to learn an optimization algorithm. We approach this problem from a reinforcement learning perspective and represent any particular optimization algorithm as a policy. We learn an optimization algorithm using guided policy search and demonstrate that the resulting algorithm outperforms existing hand-engineered algorithms in terms of convergence speed and/or the final objective value.
1 INTRODUCTION
Continuous optimization algorithms are some of the most ubiquitous tools used in virtually all areas of science and engineering. Indeed, they are the workhorse of machine learning and power most learning algorithms. Consequently, optimization difficulties become learning challenges – because their causes are often not well understood, they are one of the most vexing issues that arise in practice. One solution is to design better optimization algorithms that are immune to these failure cases. This requires careful analysis of existing optimization algorithms and clever solutions to overcome their weaknesses; thus, doing so is both laborious and time-consuming. Is there a better way? If the mantra of machine learning is to learn what is traditionally manually designed, why not take it a step further and learn the optimization algorithm itself?
Consider the general structure of an algorithm for unconstrained continuous optimization, which is outlined in Algorithm 1. Starting from a random location in the domain of the objective function, the algorithm iteratively updates the current iterate by a step vector ∆x computed from some functional π of the objective function, the current iterate and past iterates.
Algorithm 1 General structure of unconstrained optimization algorithms Require: Objective function f x(0) ← random point in the domain of f for i = 1, 2, . . . do
∆x← π(f, {x(0), . . . , x(i−1)}) if stopping condition is met then
return x(i−1) end if x(i) ← x(i−1) + ∆x
end for
Different optimization algorithms only differ in the choice of the update formula π. Examples of existing optimization algorithms and their corresponding update formulas are shown in Table 1.
If we can learn π, we will be able to learn an optimization algorithm. Since it is difficult to model general functionals, in practice, we restrict the dependence of π on the objective function f to objective values and gradients evaluated at current and past iterates. Hence, π can be simply modelled as a function from the objective values and gradients along the trajectory taken by the optimizer so far
to the next step vector. If we model π with a universal function approximator like a neural net, it is then possible to search over the space of optimization algorithms by learning the parameters of the neural net. We formulate this as a reinforcement learning problem, where any particular optimization algorithm simply corresponds to a policy. Learning an optimization algorithm then reduces to finding an optimal policy. For this purpose, we use an off-the-shelf reinforcement learning algorithm known as guided policy search (Levine & Abbeel, 2014), which has demonstrated success in a variety of robotic control settings (Levine et al., 2015a; Finn et al., 2015; Levine et al., 2015b; Han et al., 2015).
Our goal is to learn about regularities in the geometry of the error surface induced by a class of objective functions of interest and exploit this knowledge to optimize the class of objective functions faster. This is potentially advantageous, since the learned optimizer is trained on the actual objective functions that arise in practice, whereas hand-engineered optimizers are often analyzed in the convex setting and applied to the non-convex setting.
1.1 LEARNING HOW TO LEARN
When the objective functions under consideration correspond to loss functions for training a model, the proposed framework effectively learns how to learn. The loss function for training a model on a particular task/dataset is a particular objective function, and so the loss on many tasks corresponds to a set of objective functions. We can train an optimizer on this set of objective functions, which can hopefully learn to exploit regularities of the model and train it faster irrespective of the task. As a concrete example, if the model is a neural net with ReLU activations, our goal is to learn an optimizer that can leverage the piecewise linearity of the model.
We evaluate the learned optimizer on its ability to generalize to unseen objective functions. Akin to the supervised learning paradigm, we divide the dataset of objective functions into training and test sets. At test time, the learned optimizer can be used stand-alone and functions exactly like a hand-engineered optimizer, except that the update formula is replaced with a neural net and no hyperparameters like step size or momentum need to be specified by the user. In particular, it does not perform multiple trials on the same objective function at test time, unlike hyperparameter optimization. Since different objective functions correspond to the loss for training a model on different tasks, the optimizer is effectively asked to train on its experience of learning on some tasks and generalize to other possibly unrelated tasks. It is therefore critical to ensure that the optimizer does not learn anything about particular tasks; this would be considered as overfitting under this setting. Notably, this goal is different from the line of work on “learning to learn” or “meta-learning”, whose goal is to learn something about a family of tasks. To prevent overfitting to particular tasks, we train the optimizer to learn on randomly generated tasks.
2 RELATED WORK
2.1 META-LEARNING
When the objective functions optimized by the learned optimizer correspond to loss functions for training a model, learning the optimizer can be viewed as learning how to learn. This theme of learning about learning itself has been explored and is referred to as “learning to learn” or “meta-
learning” (Baxter et al., 1995; Vilalta & Drissi, 2002; Brazdil et al., 2008; Thrun & Pratt, 2012). Various authors have used the term in different ways and there is no consensus on its precise definition. While there is agreement on what kinds of knowledge should be learned at the base-level, it is less clear what kinds of meta-knowledge should be learned at the meta-level. We briefly summarize the various perspectives that have been presented below.
One form of meta-knowledge is the commonalities across a family of related tasks (Abu-Mostafa, 1993). Under this framework, the goal of the meta-learner is to learn about such commonalities, so that given the learned commonalities, base-level learning on new tasks from the family can be performed faster. This line of work is often better known as transfer learning and multi-task learning.
A different approach (Brazdil et al., 2003) is to learn how to select the base-level learner that achieves the best performance for a given task. Under this setting, the meta-knowledge is the correlation between properties of tasks and the performance of different base-level learners trained on them. There are two challenges associated with this approach: the need to devise meta-features on tasks that capture similarity between different tasks, and the need to parameterize the space of base-level learners to make search in this space tractable.
Schmidhuber (2004) proposes representing base-level learners as general-purpose programs, that is, sequences of primitive operations. While such a representation can in principle encode all base-level learners, searching in this space takes exponential time in the length of the target program.
The proposed method differs from these lines in work in several important ways. First, the proposed method learns regularities in the optimization/learning process itself, rather than regularities that are shared by different tasks or regularities in the mapping between tasks and best-performing base-level learners. More concretely, the meta-knowledge in the proposed framework can capture regularities in the error surface. Second, unlike the approaches above, we explicitly aim to avoid capturing any regularities about the task. Under the proposed framework, only model-specific regularities are captured at the meta-level, while task-specific regularities are captured at the base-level.
2.2 PROGRAM INDUCTION
Because the proposed method learns an algorithm, it is related to program induction, which considers the problem of learning programs from examples of input and output. Several different approaches have been proposed: genetic programming (Cramer, 1985) represents programs as abstract syntax trees and evolves them using genetic algorithms, Liang et al. (2010) represents programs explicitly using a formal language, constructs a hierarchical Bayesian prior over programs and performs inference using an MCMC sampling procedure and Graves et al. (2014) represents programs implicitly as sequences of memory access operations and trains a recurrent neural net to learn the underlying patterns in the memory access operations. Hochreiter et al. (2001) considers the special case of online learning algorithms, each of which is represented as a recurrent neural net with a particular setting of weights, and learns the online learning algorithm by learning the neural net weights. While the program/algorithm improves as training progresses, the algorithms learned using these methods have not been able to match the performance of simple hand-engineered algorithms. In contrast, our aim is learn an algorithm that is better than known hand-engineered algorithms.
2.3 HYPERPARAMETER OPTIMIZATION
There is a large body of work on hyperparameter optimization, which studies the optimization of hyperparameters used to train a model, such as the learning rate, the momentum decay factor and regularization parameters. Most methods (Hutter et al., 2011; Bergstra et al., 2011; Snoek et al., 2012; Swersky et al., 2013; Feurer et al., 2015) rely on sequential model-based Bayesian optimization (Mockus et al., 1978; Brochu et al., 2010), while others adopt a random search approach (Bergstra & Bengio, 2012) or use gradient-based optimization (Bengio, 2000; Domke, 2012; Maclaurin et al., 2015). Because each hyperparameter setting corresponds to a particular instantiation of an optimization algorithm, these methods can be viewed as a way to search over different instantiations of the same optimization algorithm. The proposed method, on the other hand, can search over a larger space of possible optimization algorithms. In addition, as noted previously, when presented with a new objective function at test time, the learned optimizer does not need to conduct multiple trials with different hyperparameter settings.
2.4 SPECIAL CASES AND OTHER RELATED WORK
Work on online hyperparameter adaptation studies ways to choose the step size or other hyperparameters adaptively while performing optimization. Stochastic meta-descent (Bray et al., 2004) derives a rule for adaptively choosing the step size, Ruvolo et al. (2009) learns a policy for picking the damping factor in the Levenberg-Marquardt algorithm and recent work (Hansen, 2016; Daniel et al., 2016; Fu et al., 2016) explores learning a policy for choosing the step size. Unlike this line of work, the proposed method learns a policy for choosing the step direction as well as step size, thereby making it possible to learn a new optimization algorithm that is different from known algorithms. A different line of work (Gregor & LeCun, 2010; Sprechmann et al., 2013) explores learning special-purpose solvers for a class of optimization problems that arise in sparse coding.
Work that appeared on ArXiv after this paper (Andrychowicz et al., 2016) explores a similar theme under a different setting, where the goal is learn a task-dependent optimization algorithm. The optimizer is trained from the experience of training on a particular task or family of tasks and is evaluated on its ability to train on the same task or family of tasks. Under this setting, the optimizer learns regularities about the task itself rather than regularities of the model in general.
3 BACKGROUND ON REINFORCEMENT LEARNING
3.1 MARKOV DECISION PROCESS
In the reinforcement learning setting, the learner is given a choice of actions to take in each time step, which changes the state of the environment in an unknown fashion, and receives feedback based on the consequence of the action. The feedback is typically given in the form of a reward or cost, and the objective of the learner is to choose a sequence of actions based on observations of the current environment that maximizes cumulative reward or minimizes cumulative cost over all time steps.
More formally, a reinforcement learning problem can be characterized by a Markov decision process (MDP). We consider an undiscounted finite-horizon MDP with continuous state and action spaces defined by the tuple (S,A, p0, p, c, T ), where S ⊆ RD is the set of states, A ⊆ Rd is the set of actions, p0 : S → R+ is the probability density over initial states, p : S × A × S → R+ is the transition probability density, that is, the conditional probability density over successor states given the current state and action, c : S → R is a function that maps state to cost and T is the time horizon. A policy π : S × A × {0, . . . , T − 1} → R+ is a conditional probability density over actions given the state at each time step. When a policy is independent of the time step, it is referred to as stationary.
3.2 POLICY SEARCH
This problem of finding the cost-minimizing policy is known as the policy search problem. More precisely, the objective is to find a policy π∗ such that
π∗ = arg min π Es0,a0,s1,...,sT [ T∑ t=0 c(st) ] ,
where the expectation is taken with respect to the joint distribution over the sequence of states and actions, often referred to as a trajectory, which has the density
q (s0, a0, s1, . . . , sT ) = p0 (s0) T−1∏ t=0 π (at| st, t) p (st+1| st, at) .
To enable generalization to unseen states, the policy is typically parameterized and minimization is performed over representable policies. Solving this problem exactly is intractable in all but selected special cases. Therefore, policy search methods generally tackle this problem by solving it approximately. In addition, the transition probability density p is typically not known, but may be accessed via sampling.
3.3 GUIDED POLICY SEARCH
Guided policy search (GPS) (Levine & Abbeel, 2014) is a method for searching over expressive non-linear policy classes in continuous state and action spaces. It works by alternating between computing a mixture of target trajectories and training the policy to replicate them. Successive iterations locally improve target trajectories while ensuring proximity to behaviours that are reproducible by the policy. Target trajectories are computed by fitting local approximations to the cost and transition probability density and optimizing over a restricted class of time-varying linear target policies subject to a trust region constraint. The stationary non-linear policy is trained to minimize the squared Mahalanobis distance between the predicted and target actions at each time step.
More precisely, GPS works by solving the following constrained optimization problem:
min θ,η
Eψ [ T∑ t=0 c(st) ] s.t. ψ (at| st, t; η) = π (at| st; θ) ∀at, st, t,
where ψ denotes the time-varying target policy, π denotes the stationary non-linear policy, and Eψ [·] denotes the expectation taken with respect to the trajectory induced by the target policy ψ. ψ is assumed to be conditionally Gaussian whose mean is linear in st and π is assumed to be conditionally Gaussian whose mean could be an arbitrary function of st. To solve this problem, the equality constraint is relaxed and replaced with a penalty on the KL-divergence between ψ and π. Different flavours of GPS (Levine & Abbeel, 2014; Levine et al., 2015a) use different constrained optimization methods, which all involve alternating between optimizing the parameters of ψ and π.
For updating ψ, GPS first builds a model p̃ of the transition probability density p of the form p̃ (st+1| st, at, t) := N (Atst + Btat + ct, Ft)1, where At, Bt and ct are parameters estimated from samples drawn from the trajectory induced by the existing ψ. It also computes local quadratic approximations to the cost, so that c(st) ≈ 12s T t Ctst + d T t st + ht for st’s that are near the samples. It then solves the following:
min Kt,kt,Gt Eψ̃ [ T∑ t=0 1 2 sTt Ctst + d T t st ]
s.t. T∑ t=0 DKL (p (st)ψ ( ·| st, t; η)‖ p (st)ψ ( ·| st, t; η′)) ≤ ,
where Eψ̃ [·] denotes the expectation taken with respect to the trajectory induced by the target policy ψ if states transition according to the model p̃. Kt, kt, Gt are the parameters of ψ (at| st, t; η) := N (Ktst + kt, Gt) and η′ denotes the parameters of the previous target policy. It turns out that this optimization problem can be solved in closed form using a dynamic programming algorithm known as linear-quadratic-Gaussian regulator (LQG).
For updating π, GPS minimizes DKL (p (st)π ( ·| st)‖ p (st)ψ ( ·| st, t)). Assuming fixed covariance and omitting dual variables, this corresponds to minimizing the following:
Eψ [ T∑ t=0 (Eπ [at| st]− Eψ [at| st, t])T G−1t (Eπ [at| st]− Eψ [at| st, t]) ] ,
where Eπ [·] denotes the expectation taken with respect to the trajectory induced by the non-linear policy π. We refer interested readers to (Levine & Abbeel, 2014) and (Levine et al., 2015a) for details.
4 FORMULATION
We observe that the execution of an optimization algorithm can be viewed as the execution of a particular policy in an MDP: the state consists of the current iterate and the objective values and gradients evaluated at the current and past iterates, the action is the step vector that is used to update
1In a slight abuse of notation, we use N (µ,Σ) to denote the density of a Gaussian distribution with mean µ and covariance Σ.
the current iterate, and the transition probability is partially characterized by the update formula, x(i) ← x(i−1) + ∆x. The policy that is executed corresponds precisely to the choice of π used by the optimization algorithm. For this reason, we will also use π to denote the policy at hand. Under this formulation, searching over policies corresponds to searching over possible optimization algorithms.
To learn π, we need to define the cost function, which should penalize policies that exhibit undesirable behaviours during their execution. Since the performance metric of interest for optimization algorithms is the speed of convergence, the cost function should penalize policies that converge slowly. To this end, assuming the goal is to minimize the objective function, we define cost at a state to be the objective value at the current iterate. This encourages the policy to reach the minimum of the objective function as quickly as possible. We choose to parameterize the mean of π using a neural net, due to its appealing properties as a universal function approximator and strong empirical performance in a variety of applications. We use GPS to learn π.
5 IMPLEMENTATION DETAILS
We store the current iterate, previous gradients and improvements in the objective value from previous iterations in the state. We keep track of only the information pertaining to the previous H time steps and use H = 25 in our experiments. More specifically, the dimensions of the state space encode the following information:
• Current iterate
• Change in the objective value at the current iterate relative to the objective value at the ith most recent iterate for all i ∈ {2, . . . ,H + 1}
• Gradient of the objective function evaluated at the ith most recent iterate for all i ∈ {2, . . . ,H + 1}
Initially, we set the dimensions corresponding to historical information to zero. The current iterate is only used to compute the cost; because the policy should not depend on the absolute coordinates of the current iterate, we exclude it from the input that is fed into the neural net.
We use a small neural net with a single hidden layer of 50 hidden units to model the mean of π. Softplus activation units are used at the hidden layer and linear activation units are used at the output layer. We initialize the weights of the neural net randomly and do not regularize the magnitude of weights.
Initially, we set the target trajectory distribution so that the mean action given state at each time step matches the step vector used by the gradient descent method with momentum. We choose the best settings of the step size and momentum decay factor for each objective function in the training set by performing a grid search over hyperparameters and running noiseless gradient descent with momentum for each hyperparameter setting. We use a mixture of 10 Gaussians as a prior for fitting the parameters of the transition probability density.
For training, we sample 20 trajectories with a length of 40 time steps for each objective function in the training set. After each iteration of guided policy search, we sample new trajectories from the new distribution and discard the trajectories from the preceding iteration.
6 EXPERIMENTS
We learn optimization algorithms for various convex and non-convex classes of objective functions that correspond to loss functions for different machine learning models. We learn an optimizer for logistic regression, robust linear regression using the Geman-McClure M-estimator and a twolayer neural net classifier with ReLU activation units. The geometry of the error surface becomes progressively more complex: the loss for logistic regression is convex, the loss for robust linear regression is non-convex, and the loss for the neural net has many local minima.
6.1 LOGISTIC REGRESSION
We consider a logistic regression model with an `2 regularizer on the weight vector. Training the model requires optimizing the following objective:
min w,b − 1 n n∑ i=1 yi log σ ( wTxi + b ) + (1− yi) log ( 1− σ ( wTxi + b )) + λ 2 ‖w‖22 ,
where w ∈ Rd and b ∈ R denote the weight vector and bias respectively, xi ∈ Rd and yi ∈ {0, 1} denote the feature vector and label of the ith instance, λ denotes the coefficient on the regularizer and σ(z) := 11+e−z . For our experiments, we choose λ = 0.0005 and d = 3. This objective is convex in w and b.
We train an algorithm for optimizing objectives of this form. Different examples in the training set correspond to such objective functions with different instantiations of the free variables, which in this case are xi and yi. Hence, each objective function in the training set corresponds to a logistic regression problem on a different dataset.
To construct the training set, we randomly generate a dataset of 100 instances for each function in the training set. The instances are drawn randomly from two multivariate Gaussians with random means and covariances, with half drawn from each. Instances from the same Gaussian are assigned the same label and instances from different Gaussians are assigned different labels.
We train the optimizer on a set of 90 objective functions. We evaluate it on a test set of 100 random objective functions generated using the same procedure and compare to popular hand-engineered algorithms, such as gradient descent, momentum, conjugate gradient and L-BFGS. All baselines are run with the best hyperparameter settings tuned on the training set.
For each algorithm and objective function in the test set, we compute the difference between the objective value achieved by a given algorithm and that achieved by the best of the competing algorithms at every iteration, a quantity we will refer to as “the margin of victory”. This quantity is positive when the current algorithm is better than all other algorithms and negative otherwise. In Figure 1a, we plot the mean margin of victory of each algorithm at each iteration averaged over all objective functions in the test set.
As shown, the learned optimizer, which we will henceforth refer to as “predicted step descent”, outperforms gradient descent, momentum and conjugate gradient at almost every iteration. The margin of victory for predicted step descent is high in early iterations, indicating that it converges much faster than other algorithms. It is interesting to note that despite having seen only trajectories of length 40 at training time, the learned optimizer is able to generalize to much longer time horizons at test time. L-BFGS converges to slightly better optima than predicted step descent and the momentum method. This is not surprising, as the objective functions are convex and L-BFGS is known to be a very good optimizer for convex problems.
We show the performance of each algorithm on two objective functions from the test set in Figures 1b and 1c. In Figure 1b, predicted step descent converges faster than all other algorithms. In Figure 1c, predicted step descent initially converges faster than all other algorithms but is later overtaken by L-BFGS, while remaining faster than all other optimizers. However, it eventually achieves the same objective value as L-BFGS, while the objective values achieved by gradient descent and momentum remain much higher.
6.2 ROBUST LINEAR REGRESSION
Next, we consider the problem of linear regression using a robust loss function. One way to ensure robustness is to use an M-estimator for parameter estimation. A popular choice is the GemanMcClure estimator, which induces the following objective:
min w,b
1
n n∑ i=1
( yi −wTxi − b )2 c2 + (yi −wTxi − b)2 ,
where w ∈ Rd and b ∈ R denote the weight vector and bias respectively, xi ∈ Rd and yi ∈ R denote the feature vector and label of the ith instance and c ∈ R is a constant that modulates the shape of the loss function. For our experiments, we use c = 1 and d = 3. This loss function is not convex in either w or b.
As with the preceding section, each objective function in the training set is a function of the above form with a particular instantiation of xi and yi. The dataset for each objective function is generated by drawing 25 random samples from each one of four multivariate Gaussians, each of which has a random mean and the identity covariance matrix. For all points drawn from the same Gaussian, their labels are generated by projecting them along the same random vector, adding the same randomly generated bias and perturbing them with i.i.d. Gaussian noise.
The optimizer is trained on a set of 120 objective functions. We evaluate it on 100 randomly generated objective functions using the same metric as above. As shown in Figure 2a, predicted step descent outperforms all hand-engineered algorithms except at early iterations. While it dominates gradient descent, conjugate gradient and L-BFGS at all times, it does not make progress as quickly as the momentum method initially. However, after around 30 iterations, it is able to close the gap and surpass the momentum method. On this optimization problem, both conjugate gradient and L-BFGS diverge quickly. Interestingly, unlike in the previous experiment, L-BFGS no longer performs well, which could be caused by non-convexity of the objective functions.
Figures 2b and 2c show performance on objective functions from the test set. In Figure 2b, predicted step descent not only converges the fastest, but also reaches a better optimum than all other algorithms. In Figure 2c, predicted step descent converges the fastest and is able to avoid most of the oscillations that hamper gradient descent and momentum after reaching the optimum.
6.3 NEURAL NET CLASSIFIER
Finally, we train an optimizer to train a small neural net classifier. We consider a two-layer neural net with ReLU activation on the hidden units and softmax activation on the output units. We use the cross-entropy loss combined with `2 regularization on the weights. To train the model, we need to optimize the following objective:
min W,U,b,c − 1 n n∑ i=1 log
 exp ( (U max (Wxi + b, 0) + c)yi ) ∑ j exp ( (U max (Wxi + b, 0) + c)j ) + λ 2 ‖W‖2F + λ 2 ‖U‖2F ,
where W ∈ Rh×d, b ∈ Rh, U ∈ Rp×h, c ∈ Rp denote the first-layer and second-layer weights and biases, xi ∈ Rd and yi ∈ {1, . . . , p} denote the input and target class label of the ith instance, λ denotes the coefficient on regularizers and (v)j denotes the j
th component of v. For our experiments, we use λ = 0.0005 and d = h = p = 2. The error surface is known to have complex geometry and multiple local optima, making this a challenging optimization problem.
The training set consists of 80 objective functions, each of which corresponds to the objective for training a neural net on a different dataset. Each dataset is generated by generating four multivariate Gaussians with random means and covariances and sampling 25 points from each. The points from the same Gaussian are assigned the same random label of either 0 or 1. We make sure not all of the points in the dataset are assigned the same label.
We evaluate the learned optimizer in the same manner as above. As shown in Figure 3a, predicted step descent significantly outperforms all other algorithms. In particular, as evidenced by the sizeable and sustained gap between margin of victory for predicted step descent and the momentum method, predicted step descent is able to reach much better optima and is less prone to getting trapped in local optima compared to other methods. This gap is also larger compared to that exhibited in previous sections, suggesting that hand-engineered algorithms are more sub-optimal on challenging optimization problems and so the potential for improvement from learning the algorithm is greater in such settings. Due to non-convexity, conjugate gradient and L-BFGS often diverge.
Performance on examples of objective functions from the test set is shown in Figures 3b and 3c. As shown, predicted step descent is able to reach better optima than all other methods and largely avoids oscillations that other methods suffer from.
6.4 VISUALIZATION OF OPTIMIZATION TRAJECTORIES
We visualize optimization trajectories followed by the learned algorithm and various handengineered algorithms to gain further insights into the behaviour of the learned algorithm. We generated random two-dimensional logistic regression problems and plot trajectories followed by different algorithms on each problem in Figure 4.
As shown, the learned algorithm exhibits some interesting behaviours. In Figure 4a, the learned algorithm does not take as large a step as L-BFGS initially, but takes larger steps than L-BFGS later on as it approaches the optimum. In other words, the learned algorithm appears to be not as greedy as L-BFGS. In Figures 4b and 4d, the learned algorithm initially overshoots, but appears to have learned how to recover while avoiding oscillations. In Figure 4c, the learned algorithm is able to make rapid progress despite vanishing gradients.
7 CONCLUSION
We presented a method for learning a better optimization algorithm. We formulated this as a reinforcement learning problem, in which any particular optimization algorithm can be represented as a policy. Learning an optimization algorithm then reduces to find the optimal policy. We used guided policy search for this purpose and trained optimizers for different classes of convex and non-convex objective functions. We demonstrated that the learned optimizer converges faster and/or reaches better optima than hand-engineered optimizers. We hope optimizers learned using the proposed approach can be used to solve various common classes of optimization problems more quickly and help accelerate the pace of research in science and engineering.
ACKNOWLEDGEMENTS
This work was supported by ONR MURI N00014-14-1-0671. Ke Li thanks the Natural Sciences and Engineering Research Council of Canada (NSERC) for fellowship support. The authors also thank Chelsea Finn for code and Pieter Abbeel, Sandy Huang and Zoe McCarthy for feedback. This research used the Savio computational cluster resource provided by the Berkeley Research Computing program at the University of California, Berkeley (supported by the UC Berkeley Chancellor, Vice Chancellor for Research, and Chief Information Officer).
8 APPENDIX
8.1 TRANSFER TO OBJECTIVE FUNCTIONS DRAWN FROM DIFFERENT DISTRIBUTIONS
We evaluate the learned neural net optimizer, which is trained on neural net classification problems on data drawn from mixtures of four random Gaussians, on neural net classification problems on data drawn from mixtures of different numbers of random Gaussians. As the number of mixture components increases, the data distributions used at test time become more dissimilar from those seen during training. As shown in Figures 5, the learned optimizer seems to be fairly robust and performs reasonably well despite deviations of data distributions from those used for training.

This paper proposes an approach to learning a custom optimizer for a given class optimization problems. I think in the case of training machine learning algorithms, a class would represent a model like “logistic regression”. The authors cleverly cast this as a reinforcement learning problem and use guided policy search to train a neural network to map the current location and history onto a step direction/magnitude. Overall I think this is a great idea and a very nice contribution to the fast growing meta-learning literature. However, I think that there are some aspects that could be touched on to make this a stronger paper.
My first thought is that the authors claim to train the method to learn the regularities of an entire class of optimization problems, rather than learning to exploit regularities in a given set of tasks. The distinction here is not terribly clear to me. For example, in learning an optimizer for logistic regression, the authors seem to claim that learning on a randomly sampled set of logistic regression problems will allow the model to learn about logistic regression itself. I am not convinced of this, because there is bias in the randomly sampled data itself. From the paper in this case, “The instances are drawn randomly from two multivariate Gaussians with random means and covariances, with half drawn from each.” It seems the optimizer is then trained to optimize instances of logistic regression *given this specific family of training inputs* and not logistic regression problems in general. A simple experiment to prove the method works more generally would be to repeat the existing experiments, but where the test instances are drawn from a completely different distribution. It would be even more interesting to see how this changes as the test distribution deviates further from the training distribution.
Can the authors comment on the choice of architecture used here? Why one layer with 50 hidden units and softplus activations specifically? Why not e.g., 100 units, 2 layers and ReLUs?
Presumably this is to prevent overfitting, but given the limited capacity of the network, how do these results look when the dimensionality of the input space increases beyond 2 or 3?
I would love to see what kind of policy the network learns on e.g., a 2D function using a contour plot. What do the steps look like on a random problem instance when compared to other hand-engineered optimizers?
Overall I think this a really interesting paper with a great methodological contribution. My main concern is that the results may be oversold as the problems are still relatively simple and constrained. However, if the authors can demonstrate that this approach produces robust policies for a very general set of problems then that would be truly spectacular.
Minor notes below.
Section 3.1 should you be using \pi_T^* to denote the optimal policy? You use \pi_t^* and \pi^* currently.
Are the problems here considered noiseless? That is, is the state transition given an action deterministic? It would be very interesting to see this on noisy problems.


A COMPARE-AGGREGATE MODEL FOR MATCHING TEXT SEQUENCES
Many NLP tasks including machine comprehension, answer selection and text entailment require the comparison between sequences. Matching the important units between sequences is a key to solve these problems. In this paper, we present a general “compare-aggregate” framework that performs word-level matching followed by aggregation using Convolutional Neural Networks. We particularly focus on the different comparison functions we can use to match two vectors. We use four different datasets to evaluate the model. We find that some simple comparison functions based on element-wise operations can work better than standard neural network and neural tensor network.
1 INTRODUCTION
Many natural language processing problems involve matching two or more sequences to make a decision. For example, in textual entailment, one needs to determine whether a hypothesis sentence can be inferred from a premise sentence (Bowman et al., 2015). In machine comprehension, given a passage, a question needs to be matched against it in order to find the correct answer (Richardson et al., 2013; Tapaswi et al., 2016). Table 1 gives two example sequence matching problems. In the first example, a passage, a question and four candidate answers are given. We can see that to get the correct answer, we need to match the question against the passage and identify the last sentence to be the answer-bearing sentence. In the second example, given a question and a set of candidate answers, we need to find the answer that best matches the question. Because of the fundamental importance of comparing two sequences of text to judge their semantic similarity or relatedness, sequence matching has been well studied in natural language processing.
With recent advances of neural network models in natural language processing, a standard practice for sequence modeling now is to encode a sequence of text as an embedding vector using models such as RNN and CNN. To match two sequences, a straightforward approach is to encode each sequence as a vector and then to combine the two vectors to make a decision (Bowman et al., 2015; Feng et al., 2015). However, it has been found that using a single vector to encode an entire sequence is not sufficient to capture all the important information from the sequence, and therefore advanced techniques such as attention mechanisms and memory networks have been applied to sequence matching problems (Hermann et al., 2015; Hill et al., 2016; Rocktäschel et al., 2015).
A common trait of a number of these recent studies on sequence matching problems is the use of a “compare-aggregate” framework (Wang & Jiang, 2016b; He & Lin, 2016; Parikh et al., 2016). In such a framework, comparison of two sequences is not done by comparing two vectors each representing an entire sequence. Instead, these models first compare vector representations of smaller units such as words from these sequences and then aggregate these comparison results to make the final decision. For example, the match-LSTM model proposed by Wang & Jiang (2016b) for textual entailment first compares each word in the hypothesis with an attention-weighted version of the premise. The comparison results are then aggregated through an LSTM. He & Lin (2016) proposed a pairwise word interaction model that first takes each pair of words from two sequences and applies a comparison unit on the two words. It then combines the results of these word interactions using a similarity focus layer followed by a multi-layer CNN. Parikh et al. (2016) proposed a decomposable attention model for textual entailment, in which words from each sequence are compared with an
attention-weighted version of the other sequence to produce a series of comparison vectors. The comparison vectors are then aggregated and fed into a feed forward network for final classification.
Although these studies have shown the effectiveness of such a “compare-aggregate” framework for sequence matching, there are at least two limitations with these previous studies: (1) Each of the models proposed in these studies is tested on one or two tasks only, but we hypothesize that this general framework is effective on many sequence matching problems. There has not been any study that empirically verifies this. (2) More importantly, these studies did not pay much attention to the comparison function that is used to compare two small textual units. Usually a standard feedforward network is used (Hu et al., 2014; Wang & Jiang, 2016b) to combine two vectors representing two units that need to be compared, e.g., two words. However, based on the nature of these sequence matching problems, we essentially need to measure how semantically similar the two sequences are. Presumably, this property of these sequence matching problems should guide us in choosing more appropriate comparison functions. Indeed He & Lin (2016) used cosine similarity, Euclidean distance and dot product to define the comparison function, which seem to be better justifiable. But they did not systematically evaluate these similarity or distance functions or compare them with a standard feedforward network.
In this paper, we argue that the general “compare-aggregate” framework is effective for a wide range of sequence matching problems. We present a model that follows this general framework and test it on four different datasets, namely, MovieQA, InsuranceQA, WikiQA and SNLI. The first three datasets are for Question Answering, but the setups of the tasks are quite different. The last dataset is for textual entailment. More importantly, we systematically present and test six different comparison functions. We find that overall a comparison function based on element-wise subtraction and multiplication works the best on the four datasets.
The contributions of this work are twofold: (1) Using four different datasets, we show that our model following the “compare-aggregate” framework is very effective when compared with the state-ofthe-art performance on these datasets. (2) We conduct systematic evaluation of different comparison functions and show that a comparison function based on element-wise operations, which is not widely used for word-level matching, works the best across the different datasets. We believe that these findings will be useful for future research on sequence matching problems. We have also made our code available online.1
2 METHOD
In this section, we propose a general model following the “compare-aggregate” framework for matching two sequences. This general model can be applied to different tasks. We focus our discussion on six different comparison functions that can be plugged into this general “compare-aggregate” model. In particular, we hypothesize that two comparison functions based on element-wise operations, SUB and MULT, are good middle ground between highly flexible functions using standard neural network models and highly restrictive functions based on cosine similarity and/or Euclidean
1https://github.com/shuohangwang/SeqMatchSeq
distance. As we will show in the experiment section, these comparison functions based on elementwise operations can indeed perform very well on a number of sequence matching problems.
2.1 PROBLEM DEFINITION AND MODEL OVERVIEW
The general setup of the sequence matching problem we consider is the following. We assume there are two sequences to be matched. We use two matrices Q ∈ Rd×Q and A ∈ Rd×A to represent the word embeddings of the two sequences, where Q and A are the lengths of the two sequences, respectively, and d is the dimensionality of the word embeddings. In other words, each column vector of Q or A is an embedding vector representing a single word. Given a pair of Q and A, the goal is to predict a label y. For example, in textual entailment, Q may represent a premise and A a hypothesis, and y indicates whether Q entails A or contradicts A. In question answering, Q may be a question and A a candidate answer, and y indicates whether A is the correct answer to Q.
We treat the problem as a supervised learning task. We assume that a set of training examples in the form of (Q,A, y) is given and we aim to learn a model that maps any pair of (Q,A) to a y.
An overview of our model is shown in Figure 1. The model can be divided into the following four layers:
1. Preprocessing: We use a preprocessing layer (not shown in the figure) to process Q and A to obtain two new matrices Q ∈ Rl×Q and A ∈ Rl×A. The purpose here is to use some gate values to control the importance of different words in making the predictions on the sequence pair. For example, qi ∈ Rl, which is the ith column vector of Q, encodes the ith word in Q.
2. Attention: We apply a standard attention mechanism on Q and A to obtain attention weights over the column vectors in Q for each column vector in A. With these attention weights, for each column vector aj in A, we obtain a corresponding vector hj , which is an attention-weighted sum of the column vectors of Q.
3. Comparison: We use a comparison function f to combine each pair of aj and hj into a vector tj .
4. Aggregation: We use a CNN layer to aggregate the sequence of vectors tj for the final classification.
Although this model follows more or less the same framework as the model proposed by Parikh et al. (2016), our work has some notable differences. First, we will pay much attention to the comparison function f and compare a number of options, including some uncommon ones based on elementwise operations. Second, we apply our model to four different datasets representing four different tasks to evaluate its general effectiveness for sequence matching problems. There are also some other differences from the work by Parikh et al. (2016). For example, we use a CNN layer instead of summation and concatenation for aggregation. Our attention mechanism is one-directional instead of two-directional.
In the rest of this section we will present the model in detail. We will focus mostly on the comparison functions we consider.
2.2 PREPROCESSING AND ATTENTION
Inspired by the use of gates in LSTM and GRU, we preprocess Q and A with the following formulas:
Q = σ(WiQ+ bi ⊗ eQ) tanh(WuQ+ bu ⊗ eQ), A = σ(WiA+ bi ⊗ eA) tanh(WuA+ bu ⊗ eA), (1)
where is element-wise multiplication, and Wi,Wu ∈ Rl×d and bi,bu ∈ Rl are parameters to be learned. The outer product (· ⊗ eX) produces a matrix or row vector by repeating the vector or scalar on the left for X times. Here σ(WiQ + bi ⊗ eQ) and σ(WiA + bi ⊗ eA) act as gate values to control the degree to which the original values of Q and A are preserved in Q and A. For example, for stop words, their gate values would likely be low for tasks where stop words make little difference to the final predictions.
In this preprocessing step, the word order does not matter. Although a better way would be to use RNN such as LSTM and GRU to chain up the words such that we can capture some contextual information, this could be computationally expensive for long sequences. In our experiments, we only incorporated LSTM into the formulas above for the SNLI task.
The general attention (Luong et al., 2015) layer is built on top of the resulting Q and A as follows: G = softmax ( (WgQ+ bg ⊗ eQ)TA ) ,
H = QG, (2)
where Wg ∈ Rl×l and bg ∈ Rl are parameters to be learned, G ∈ RQ×A is the attention weight matrix, and H ∈ Rl×A are the attention-weighted vectors. Specifically, hj , which is the jth column vector of H, is a weighted sum of the column vectors of Q and represents the part of Q that best matches the jth word in A. Next we will combine hj and aj using a comparison function.
2.3 COMPARISON
The goal of the comparison layer is to match each aj , which represents the jth word and its context in A, with hj , which represents a weighted version of Q that best matches aj . Let f denote a comparison function that transforms aj and hj into a vector tj to represent the comparison result.
A natural choice of f is a standard neural network layer that consists of a linear transformation followed by a non-linear activation function. For example, we can consider the following choice:
NEURALNET (NN): tj = f(aj ,hj) = ReLU(W [ aj hj ] + b), (3)
where matrix W ∈ Rl×2l and vector b ∈ Rl are parameters to be learned. Alternatively, another natural choice is a neural tensor network (Socher et al., 2013) as follows:
NEURALTENSORNET (NTN): tj = f(aj ,hj) = ReLU(aTjT [1...l]hj + b), (4)
where tensor T[1...l] ∈ Rl×l×l and vector b ∈ Rl are parameters to be learned.
However, we note that for many sequence matching problems, we intend to measure the semantic similarity or relatedness of the two sequences. So at the word level, we also intend to check how similar or related aj is to hj . For this reason, a more natural choice used in some previous work is Euclidean distance or cosine similarity between aj and hj . We therefore consider the following definition of f :
EUCLIDEAN+COSINE (EUCCOS): tj = f(aj ,hj) = [ ‖aj − hj‖2 cos(aj ,hj) ] . (5)
Note that with EUCCOS, the resulting vector tj is only a 2-dimensional vector. Although EUCCOS is a well-justified comparison function, we suspect that it may lose some useful information from the original vectors aj and hj . On the other hand, NN and NTN are too general and thus do not capture the intuition that we care mostly about the similarity between aj and hj .
To use something that is a good compromise between the two extreme cases, we consider the following two new comparison functions, which operate on the two vectors in an element-wise manner. These functions have been used previously by Mou et al. (2016).
SUBTRACTION (SUB): tj = f(aj ,hj) = (aj − hj) (aj − hj), (6) MULTIPLICATION (MULT): tj = f(aj ,hj) = aj hj . (7)
Note that the operator is element-wise multiplication. For both comparison functions, the resulting vector tj has the same dimensionality as aj and hj .
We can see that SUB is closely related to Euclidean distance in that Euclidean distance is the sum of all the entries of the vector tj produced by SUB. But by not summing up these entries, SUB preserves some information about the different dimensions of the original two vectors. Similarly, MULT is closely related to cosine similarity but preserves some information about the original two vectors.
Finally, we consider combining SUB and MULT followed by an NN layer as follows: SUBMULT+NN: tj = f(aj ,hj) = ReLU(W [ (aj − hj) (aj − hj)
aj hj
] + b). (8)
In summary, we consider six different comparison functions: NN, NTN, EUCCOS, SUB, MULT and SUBMULT+NN. Among these functions, the last three (SUB, MULT and SUBMULT+NN) have not been widely used in previous work for word-level matching.
2.4 AGGREGATION
After we apply the comparison function to each pair of aj and hj to obtain a series of vectors tj , finally we aggregate these vectors using a one-layer CNN (Kim, 2014):
r = CNN([t1, . . . , tA]). (9)
r ∈ Rnl is then used for the final classification, where n is the number of windows in CNN.
3 EXPERIMENTS
In this section, we evaluate our model on four different datasets representing different tasks. The first three datasets are question answering tasks while the last one is on textual entailment. The statistics of the four datasets are shown in Table 2. We will fist introduce the task settings and the way we customize the “compare-aggregate” structure to each task. Then we will show the baselines for the different datasets. Finally, we discuss the experiment results shown in Table 3 and the ablation study shown in Table 4.
3.1 TASK-SPECIFIC MODEL STRUCTURES
In all these tasks, we use matrix Q ∈ Rd×Q to represent the question or premise and matrix Ak ∈ Rd×Ak (k ∈ [1,K]) to represent the kth answer or the hypothesis. For the machine comprehension task MovieQA (Tapaswi et al., 2016), there is also a matrix P ∈ Rd×P that represents the plot of a movie. Here Q is the length of the question or premise, Ak the length of the kth answer, and P the length of the plot.
For the SNLI (Bowman et al., 2015) dataset, the task is text entailment, which identifies the relationship (entailment, contradiction or neutral) between a premise sentence and a hypothesis sentence. Here K = 1, and there are exactly two sequences to match. The actual model structure is what we have described before.
For the InsuranceQA (Feng et al., 2015) dataset, the task is an answer selection task which needs to select the correct answer for a question from a candidate pool. For the WikiQA (Yang et al., 2015) datasets, we need to rank the candidate answers according to a question. For both tasks,
there are K candidate answers for each question. Let us use rk to represent the resulting vector produced by Eqn. 9 for the kth answer. In order to select one of the K answers, we first define R = [r1, r2, . . . , rK ]. We then compute the probability of the kth answer to be the correct one as follows:
p(k|R) = softmax(wT tanh(WsR+ bs ⊗ eK) + b⊗ eK), (10) where Ws ∈ Rl×nl, w ∈ Rl, bs ∈ Rl, b ∈ R are parameters to be learned. For the machine comprehension task MovieQA, each question is related to Plot Synopses written by fans after watching the movie and each question has five candidate answers. So for each candidate answer there are three sequences to be matched: the plot P, the question Q and the answer Ak. For each k, we first match Q and P and refer to the matching result at position j as tqj , as generated by one of the comparison functions f . Similarly, we also match Ak with P and refer to the matching result at position j as tak,j . We then define
tk,j = [ tqj tak,j ] ,
and
rk = CNN([tk,1, . . . , tk,P ]).
To select an answer from the K candidate answers, again we use Eqn. 10 to compute the probabilities.
The implementation details of the modes are as follows. The word embeddings are initialized from GloVe (Pennington et al., 2014). During training, they are not updated. The word embeddings not found in GloVe are initialized with zero.
The dimensionality l of the hidden layers is set to be 150. We use ADAMAX (Kingma & Ba, 2015) with the coefficients β1 = 0.9 and β2 = 0.999 to optimize the model. We do not use L2regularization. The main parameter we tuned is the dropout on the embedding layer. For WikiQA, which is a relatively small dataset, we also tune the learning rate and the batch size. For the others, we set the batch size to be 30 and the learning rate 0.002.
3.2 BASELINES
Here, we will introduce the baselines for each dataset. We did not re-implement these models but simply took the reported performance for the purpose of comparison.
SNLI: •W-by-W Attention: The model by Rocktäschel et al. (2015), who first introduced attention mechanism into text entailment. • match-LSTM: The model by Wang & Jiang (2016b), which concatenates the matched words as the inputs of an LSTM. • LSTMN: Long short-term memorynetworks proposed by Cheng et al. (2016). • Decomp Attention: Another “compare-aggregate” model proposed by Parikh et al. (2016). • EBIM+TreeLSTM: The state-of-the-art model proposed by Chen et al. (2016) on the SNLI dataset.
InsuranceQA: • IR model: This model by Bendersky et al. (2010) learns the concept information to help rank the candidates. • CNN with GESD: This model by Feng et al. (2015) uses Euclidean distance and dot product between sequence representations built through convolutional neural networks to select the answer. • Attentive LSTM: Tan et al. (2016) used soft-attention mechanism to select the most important information from the candidates according to the representation of the questions. • IARNN-Occam: This model by Wang et al. (2016) adds regularization on the attention weights. • IARNN-Gate: This model by Wang et al. (2016) uses the representation of the question to build the GRU gates for each candidate answer.
WikiQA: • IARNN-Occam and IARNN-Gate as introduced before. • CNN-Cnt: This model by Yang et al. (2015) combines sentence representations built by a convolutional neural network with logistic regression. • ABCNN: This model is Attention-Based Convolutional Neural Network proposed by Yin et al. (2015). • CubeCNN proposed by He & Lin (2016) builds a CNN on all pairs of word similarity.
MovieQA: All the baselines we consider come from Tapaswi et al. (2016)’s work: • Cosine Word2Vec: A sliding window is used to select the answer according to the similarities computed
through Word2Vec between the sentences in plot and the question/answer. • Cosine TFIDF: This model is similar to the previous method but uses bag-of-word with tf-idf scores to compute similarity. • SSCB TFIDF: Instead of using the sliding window method, a convolutional neural network is built on the sentence level similarities.
3.3 ANALYSIS OF RESULTS
We use accuracy as the evaluation metric for the datasets MovieQA, InsuranceQA and SNLI, as there is only one correct answer or one label for each instance. For WikiQA, there may be multiple correct answers, so evaluation metrics we use are Mean Average Precision (MAP) and Mean Reciprocal Rank (MRR).
We observe the following from the results. (1) Overall, we can find that our general “compareaggregate” structure achieves the best performance on MovieQA, InsuranceQA, WikiQA datasets and very competitive performance on the SNLI dataset. Especially for the InsuranceQA dataset, with any comparison function we use, our model can outperform all the previous models. (2) The comparison method SUBMULT+NN is the best in general. (3) Some simple comparison functions can achieve better performance than the neural networks or neural tensor network comparison functions. For example, the simplest comparison function EUCCOS achieves nearly the best performance in the MovieQA dataset, and the element-wise comparison functions, which do not need parameters can achieve the best performance on the WikiQA dataset. (4) We find the preprocessing layer and the attention layer for word selection to be important in the “compare-aggregate” structure through the experiments of removing these two layers separately. We also see that for sequence matching with big difference in length, such as the MovieQA and InsuranceQA tasks, the attention layer plays a more important role. For sequence matching with smaller difference in length, such as the WikiQA and SNLI tasks, the pre-processing layer plays a more important role. (5) For the MovieQA, InsuranceQA and WikiQA tasks, our preprocessing layer is order-insensitive so that it will not take the context information into consideration during the comparison, but our model can still outperform the previous work with order-sensitive preprocessing layer. With this finding, we believe the word-by-word comparison part plays a very important role in these tasks. We will further explore the preprocessing layer in the future.
3.4 FURTHER ANALYSES
To further explain how our model works, we visualize the max values in each dimension of the convolutional layer. We use two examples shown in Table 1 from MovieQA and InsuranceQA datasets respectively. In the top of Figure 2, we can see that the plot words that also appear in either the question or the answer will draw more attention by the CNN. We hypothesize that if the nearby words in the plot can match both the words in question and the words in one answer, then this answer is more likely to be the correct one. Similarly, the bottom one of Figure 2 also shows that the CNN will focus more on the matched word representations. If the words in one answer continuously match the words in the question, this answer is more likely to be the correct one.
4 RELATED WORK
We review related work in three types of general structures for matching sequences.
Siamense network: These kinds of models use the same structure, such as RNN or CNN, to build the representations for the sequences separately and then use them for classification. Then cosine similarity (Feng et al., 2015; Yang et al., 2015), element-wise operation (Tai et al., 2015; Mou et al., 2016) or neural network-based combination Bowman et al. (2015) are used for sequence matching.
Attentive network: Soft-attention mechanism (Bahdanau et al., 2014; Luong et al., 2015) has been widely used for sequence matching in machine comprehension (Hermann et al., 2015), text entailment (Rocktäschel et al., 2015) and question answering (Tan et al., 2016). Instead of using the final state of RNN to represent a sequence, these studies use weighted sum of all the states for the sequence representation.
Compare-Aggregate network: This kind of framework is to perform the word level matching (Wang & Jiang, 2016a; Parikh et al., 2016; He & Lin, 2016; Trischler et al., 2016; Wan et al.,
2016). Our work is under this framework. But our structure is different from previous models and our model can be applied on different tasks. Besides, we analyzed different word-level comparison functions separately.
5 CONCLUSIONS
In this paper, we systematically analyzed the effectiveness of a “compare-aggregate” model on four different datasets representing different tasks. Moreover, we compared and tested different kinds of word-level comparison functions and found that some element-wise comparison functions can outperform the others. According to our experiment results, many different tasks can share the same “compare-aggregate” structure. In the future work, we would like to test its effectiveness on multi-task learning.
6 ACKNOWLEDGMENTS
This research is supported by the National Research Foundation, Prime Ministers Office, Singapore under its International Research Centres in Singapore Funding Initiative.

The paper presents a general approach to modeling for natural language understanding problems with two distinct textual inputs (such as a question and a source text) that can be aligned in some way. In the approach, soft attention is first used to derive alignments between the tokens of the two texts, then a comparison function uses the resulting alignments (represented as pairs of attention queries and attention results) to derive a representations that are aggregated by CNN into a single vector from which an output can be computed. The paper both presents this as an overall modeling strategy that can be made to work quite well, and offers a detailed empirical analysis of the comparison component of the model.
This work is timely. Language understanding problems of this kind are a major open issue in NLP, and are just at the threshold of being addressable with representation learning methods. The work presents a general approach which is straightforward and reasonable, and shows that it can yield good results. The work borders on incremental (relative to their earlier work or that of Parikh et al.), but it contributes in enough substantial ways that I'd strongly recommend acceptance.
Detail:
- The model, at least as implemented for the problems with longer sequences (everything but SNLI), is not sensitive to word order. It is empirically competitive, but this insensitivity places a strong upper bound on its performance. The paper does make this clear, but it seems salient enough to warrant a brief mention in the introduction or discussion sections.
- If I understand correctly, your attention strategy is based more closely on the general/bilinear strategy of Luong et al. '15 than it is on the earlier Bahdanau work. You should probably cite the former (or some other more directly relevant reference for that strategy).
- Since the NTN risks overfitting because of its large number of parameters, did you try using a version with input dimension l and a smaller output dimension m (so an l*l*m tensor)?
- You should probably note that SubMultNN looks a lot like the strategy for *sentence*-level matching in the Lili Mou paper you cite.
- Is there a reason you use the same parameters for preprocessing the question and answer in (1)? These could require different things to be weighted highly.