Computation , 1(1):67–82, 1997.
9
A. S. Younger, P. R. Conwell, and N. E. Cotter. Fixed-weight on-line learning. Transactions on Neural Networks ,
10(2):272–283, 1999.
A. S. Younger, S. Hochreiter, and P. R. Conwell. Meta-learning with backpropagation. In International Joint
Conference on Neural Networks , 2001.
10
0 64 128−8−24
0 64 128−2616
LSTM
ADAM
SGDFigure 10: Updates proposed by different optimizers at different optimization steps for two different
coordinates.
A Gradient preprocessing
One potential challenge in training optimizers is that different input coordinates (i.e. the gradients
w.r.t. different optimizee parameters) can have very different magnitudes. This is indeed the case e.g.
when the optimizee is a neural network and different parameters correspond to weights in different
layers. This can make training an optimizer difﬁcult, because neural networks naturally disregard
small variations in input signals and concentrate on bigger input values.
To this aim we propose to preprocess the optimizer’s inputs. One solution would be to give the
The move from hand-designed features to learned features in machine learning
has been wildly successful. In spite of this, optimization algorithms are still
designed by hand. In this paper we show how the design of an optimization
algorithm can be cast as a learning problem, allowing the algorithm to learn to
exploit structure in the problems of interest in an automatic way. Our learned
algorithms, implemented by LSTMs, outperform generic, hand-designed competitors
on the tasks for which they are trained, and also generalize well to new tasks
with similar structure. We demonstrate this on a number of tasks, including
simple convex problems, training neural networks, and styling images with
neural art.
Marcin Andrychowicz Misha Denil Sergio Gomez Matthew W. Hoffman David Pfau Tom Schaul Brendan Shillingford Nando de Freitas