test set. The additional two plots on the right display the performance of the trained optimizer on
modiﬁed datasets which only contain a subset of the labels, i.e. the CIFAR-2 dataset only contains
data corresponding to 2 of the 10 labels. Additionally we include an optimizer LSTM-sub which was
only trained on the held-out labels.
In all these examples we can see that the LSTM optimizer learns much more quickly than the baseline
optimizers, with signiﬁcant boosts in performance for the CIFAR-5 and especially CIFAR-2 datsets.
We also see that the optimizers trained only on a disjoint subset of the data is hardly effected by this
difference and transfers well to the additional dataset.
3.4 Neural Art
The recent work on artistic style transfer using convolutional networks, or Neural Art [Gatys et al.,
2015], gives a natural testbed for our method, since each content and style image pair gives rise to a
different optimization problem. Each Neural Art problem starts from a content image ,c, and a style
image ,s, and is given by
f() =Lcontent (c;) +Lstyle(s;) +
Lreg()
The move from hand-designed features to learned features in machine learning
has been wildly successful. In spite of this, optimization algorithms are still
designed by hand. In this paper we show how the design of an optimization
algorithm can be cast as a learning problem, allowing the algorithm to learn to
exploit structure in the problems of interest in an automatic way. Our learned
algorithms, implemented by LSTMs, outperform generic, hand-designed competitors
on the tasks for which they are trained, and also generalize well to new tasks
with similar structure. We demonstrate this on a number of tasks, including
simple convex problems, training neural networks, and styling images with
neural art.
Marcin Andrychowicz Misha Denil Sergio Gomez Matthew W. Hoffman David Pfau Tom Schaul Brendan Shillingford Nando de Freitas