Gaussian distribution. Optimizers were trained by optimizing random functions from this family and
tested on newly sampled functions from the same distribution. Each function was optimized for 100
steps and the trained optimizers were unrolled for 20 steps. We have not used any preprocessing, nor
postprocessing.
Learning curves for different optimizers, averaged over many functions, are shown in the left plot of
Figure 4. Each curve corresponds to the average performance of one optimization algorithm on many
test functions; the solid curve shows the learned optimizer performance and dashed curves show
the performance of the standard baseline optimizers. It is clear the learned optimizers substantially
outperform the baselines in this setting.
3.2 Training a small neural network on MNIST
In this experiment we test whether trainable optimizers can learn to optimize a small neural network
on MNIST, and also explore how the trained optimizers generalize to functions beyond those they
were trained on. To this end, we train the optimizer to optimize a base network and explore a series
of modiﬁcations to the network architecture and training procedure at test time.
5
20 40 60 80 100100LossMNIST, 40 units
ADAM
RMSprop
SGD
NAG
LSTM
20 40 60 80 100
StepsMNIST, 2 layers
The move from hand-designed features to learned features in machine learning
has been wildly successful. In spite of this, optimization algorithms are still
designed by hand. In this paper we show how the design of an optimization
algorithm can be cast as a learning problem, allowing the algorithm to learn to
exploit structure in the problems of interest in an automatic way. Our learned
algorithms, implemented by LSTMs, outperform generic, hand-designed competitors
on the tasks for which they are trained, and also generalize well to new tasks
with similar structure. We demonstrate this on a number of tasks, including
simple convex problems, training neural networks, and styling images with
neural art.
Marcin Andrychowicz Misha Denil Sergio Gomez Matthew W. Hoffman David Pfau Tom Schaul Brendan Shillingford Nando de Freitas