ADAM
RMSprop
SGD
NAG
LSTM
20 40 60 80 100
Step100MNIST
120 140 160 180 200MNIST, 200 stepsFigure 4: Comparisons between learned and hand-crafted optimizers performance. Learned optimizers are shown with solid lines and hand-crafted optimizers are shown with dashed lines. Units for the
yaxis in the MNIST plots are logits. Left: Performance of different optimizers on randomly sampled
10-dimensional quadratic functions. Center: the LSTM optimizer outperforms standard methods
training the base network on MNIST. Right: Learning curves for steps 100-200 by an optimizer
trained to optimize for 100 steps (continuation of center plot).
3 Experiments
In all experiments the trained optimizers use two-layer LSTMs with 20 hidden units in each layer.
Each optimizer is trained by minimizing Equation 3 using truncated BPTT as described in Section 2.
The minimization is performed using ADAM with a learning rate chosen by random search.
We use early stopping when training the optimizer in order to avoid overﬁtting the optimizer. After
each epoch (some ﬁxed number of learning steps) we freeze the optimizer parameters and evaluate its
performance. We pick the best optimizer (according to the ﬁnal validation loss) and report its average
The move from hand-designed features to learned features in machine learning
has been wildly successful. In spite of this, optimization algorithms are still
designed by hand. In this paper we show how the design of an optimization
algorithm can be cast as a learning problem, allowing the algorithm to learn to
exploit structure in the problems of interest in an automatic way. Our learned
algorithms, implemented by LSTMs, outperform generic, hand-designed competitors
on the tasks for which they are trained, and also generalize well to new tasks
with similar structure. We demonstrate this on a number of tasks, including
simple convex problems, training neural networks, and styling images with
neural art.
Marcin Andrychowicz Misha Denil Sergio Gomez Matthew W. Hoffman David Pfau Tom Schaul Brendan Shillingford Nando de Freitas