performance. We pick the best optimizer (according to the ﬁnal validation loss) and report its average
performance on a number of freshly sampled test problems.
We compare our trained optimizers with standard optimizers used in Deep Learning: SGD, RMSprop,
ADAM, and Nesterov’s accelerated gradient (NAG). For each of these optimizer and each problem
we tuned the learning rate, and report results with the rate that gives the best ﬁnal error for each
problem. When an optimizer has more parameters than just a learning rate (e.g. decay coefﬁcients for
ADAM) we use the default values from the optim package in Torch7. Initial values of all optimizee
parameters were sampled from an IID Gaussian distribution.
3.1 Quadratic functions
In this experiment we consider training an optimizer on a simple class of synthetic 10-dimensional
quadratic functions. In particular we consider minimizing functions of the form
f() =kWyk2
2
for different 10x10 matrices Wand 10-dimensional vectors ywhose elements are drawn from an IID
Gaussian distribution. Optimizers were trained by optimizing random functions from this family and
tested on newly sampled functions from the same distribution. Each function was optimized for 100
The move from hand-designed features to learned features in machine learning
has been wildly successful. In spite of this, optimization algorithms are still
designed by hand. In this paper we show how the design of an optimization
algorithm can be cast as a learning problem, allowing the algorithm to learn to
exploit structure in the problems of interest in an automatic way. Our learned
algorithms, implemented by LSTMs, outperform generic, hand-designed competitors
on the tasks for which they are trained, and also generalize well to new tasks
with similar structure. We demonstrate this on a number of tasks, including
simple convex problems, training neural networks, and styling images with
neural art.
Marcin Andrychowicz Misha Denil Sergio Gomez Matthew W. Hoffman David Pfau Tom Schaul Brendan Shillingford Nando de Freitas