optimizer in this setting.
In this setting the objective function f()is the cross entropy of a small MLP with parameters .
The values of fas well as the gradients @f()=@are estimated using random minibatches of 128
examples. The base network is an MLP with one hidden layer of 20 units using a sigmoid activation
function. The only source of variability between different runs is the initial value 0and randomness
in minibatch selection. Each optimization was run for 100 steps and the trained optimizers were
unrolled for 20 steps. We used input preprocessing described in Appendix A and rescaled the outputs
of the LSTM by the factor 0:1.
Learning curves for the base network using different optimizers are displayed in the center plot of
Figure 4. In this experiment NAG, ADAM, and RMSprop exhibit roughly equivalent performance the
LSTM optimizer outperforms them by a signiﬁcant margin. The right plot in Figure 4 compares the
performance of the LSTM optimizer if it is allowed to run for 200 steps, despite having been trained
to optimize for 100 steps. In this comparison we re-used the LSTM optimizer from the previous
experiment, and here we see that the LSTM optimizer continues to outperform the baseline optimizers
on this task.
The move from hand-designed features to learned features in machine learning
has been wildly successful. In spite of this, optimization algorithms are still
designed by hand. In this paper we show how the design of an optimization
algorithm can be cast as a learning problem, allowing the algorithm to learn to
exploit structure in the problems of interest in an automatic way. Our learned
algorithms, implemented by LSTMs, outperform generic, hand-designed competitors
on the tasks for which they are trained, and also generalize well to new tasks
with similar structure. We demonstrate this on a number of tasks, including
simple convex problems, training neural networks, and styling images with
neural art.
Marcin Andrychowicz Misha Denil Sergio Gomez Matthew W. Hoffman David Pfau Tom Schaul Brendan Shillingford Nando de Freitas