experiment, and here we see that the LSTM optimizer continues to outperform the baseline optimizers
on this task.
Generalization to different architectures Figure 5 shows three examples of applying the LSTM
optimizer to train networks with different architectures than the base network on which it was trained.
The modiﬁcations are (from left to right) (1) an MLP with 40 hidden units instead of 20, (2) a
network with two hidden layers instead of one, and (3) a network using ReLU activations instead of
sigmoid. In the ﬁrst two cases the LSTM optimizer generalizes well, and continues to outperform
the hand-designed baselines despite operating outside of its training regime. However, changing
the activation function to ReLU makes the dynamics of the learning procedure sufﬁciently different
that the learned optimizer is no longer able to generalize. Finally, in Figure 6 we show the results
of systematically varying the tested architecture; for the LSTM results we again used the optimizer
trained using 1 layer of 20 units and sigmoid non-linearities. Note that in this setting where the
6
200 400 600 800 1000100LossCIFAR-10
200 400 600 800 1000
Step100CIFAR-5
200 400 600 800 100010-1CIFAR-2
The move from hand-designed features to learned features in machine learning
has been wildly successful. In spite of this, optimization algorithms are still
designed by hand. In this paper we show how the design of an optimization
algorithm can be cast as a learning problem, allowing the algorithm to learn to
exploit structure in the problems of interest in an automatic way. Our learned
algorithms, implemented by LSTMs, outperform generic, hand-designed competitors
on the tasks for which they are trained, and also generalize well to new tasks
with similar structure. We demonstrate this on a number of tasks, including
simple convex problems, training neural networks, and styling images with
neural art.
Marcin Andrychowicz Misha Denil Sergio Gomez Matthew W. Hoffman David Pfau Tom Schaul Brendan Shillingford Nando de Freitas