as the previous hidden state and outputs the update for the corresponding optimizee parameter.
We will refer to this architecture, illustrated in
Figure 3, as an LSTM optimizer.
The use of recurrence allows the LSTM to learn
dynamic update rules which integrate information from the history of gradients, similar to
momentum. This is known to have many desirable properties in convex optimization [see e.g.
Nesterov, 1983] and in fact many recent learning procedures—such as ADAM—use momentum in
their updates.
Preprocessing and postprocessing Optimizer inputs and outputs can have very different magnitudes depending on the class of function being optimized, but neural networks usually work robustly
only for inputs and outputs which are neither very small nor very large. In practice rescaling inputs
and outputs of an LSTM optimizer using suitable constants (shared across all timesteps and functions
f) is sufﬁcient to avoid this problem. In Appendix A we propose a different method of preprocessing
inputs to the optimizer inputs which is more robust and gives slightly better performance.
4
20 40 60 80 10010-1100101LossQuadratics
ADAM
RMSprop
SGD
NAG
LSTM
20 40 60 80 100
Step100MNIST
The move from hand-designed features to learned features in machine learning
has been wildly successful. In spite of this, optimization algorithms are still
designed by hand. In this paper we show how the design of an optimization
algorithm can be cast as a learning problem, allowing the algorithm to learn to
exploit structure in the problems of interest in an automatic way. Our learned
algorithms, implemented by LSTMs, outperform generic, hand-designed competitors
on the tasks for which they are trained, and also generalize well to new tasks
with similar structure. We demonstrate this on a number of tasks, including
simple convex problems, training neural networks, and styling images with
neural art.
Marcin Andrychowicz Misha Denil Sergio Gomez Matthew W. Hoffman David Pfau Tom Schaul Brendan Shillingford Nando de Freitas