ADAM
RMSprop
SGD
NAG
LSTM
20 40 60 80 100
StepsMNIST, 2 layers
20 40 60 80 100MNIST, ReLUFigure 5: Comparisons between learned and hand-crafted optimizers performance. Units for the
yaxis are logits. Left: Generalization to the different number of hidden units (40 instead of 20).
Center: Generalization to the different number of hidden layers (2 instead of 1). This optimization
problem is very hard, because the hidden layers are very narrow. Right: Training curves for an MLP
with 20 hidden units using ReLU activations. The LSTM optimizer was trained on an MLP with
sigmoid activations.
5 20 35 5010-1100Final Loss
LSTM
5 20 35 50
Hidden units
ADAM
5 20 35 50
NAG
Layers
1
2
Figure 6: Systematic study of ﬁnal MNIST performance as the optimizee architecture is varied,
using sigmoid non-linearities. The vertical dashed line in the left-most plot denotes the architecture
at which the LSTM is trained and the horizontal line shows the ﬁnal performance of the trained
optimizer in this setting.
In this setting the objective function f()is the cross entropy of a small MLP with parameters .
The move from hand-designed features to learned features in machine learning
has been wildly successful. In spite of this, optimization algorithms are still
designed by hand. In this paper we show how the design of an optimization
algorithm can be cast as a learning problem, allowing the algorithm to learn to
exploit structure in the problems of interest in an automatic way. Our learned
algorithms, implemented by LSTMs, outperform generic, hand-designed competitors
on the tasks for which they are trained, and also generalize well to new tasks
with similar structure. We demonstrate this on a number of tasks, including
simple convex problems, training neural networks, and styling images with
neural art.
Marcin Andrychowicz Misha Denil Sergio Gomez Matthew W. Hoffman David Pfau Tom Schaul Brendan Shillingford Nando de Freitas