will use an optimizer mwhich operates coordinatewise on the parameters of the objective function,
similar to other common update rules like RMSprop and ADAM. This coordinatewise network
architecture allows us to use a very small network that only looks at a single coordinate to deﬁne the
optimizer and share optimizer parameters across different parameters of the optimizee.
Different behavior on each coordinate is achieved by using separate activations for each objective
function parameter. In addition to allowing us to use a small network for this optimizer, this setup has
the nice effect of making the optimizer invariant to the order of parameters in the network, since the
same update rule is used independently on each coordinate.
LSTM1fLSTMn∇1θ1θn++
…∇n………
…
Figure 3: One step of an LSTM optimizer. All
LSTMs have shared parameters, but separate hidden states.We implement the update rule for each coordinate using a two-layer Long Short Term Memory
(LSTM) network [Hochreiter and Schmidhuber,
1997], using the now-standard forget gate architecture. The network takes as input the optimizee gradient for a single coordinate as well
as the previous hidden state and outputs the update for the corresponding optimizee parameter.
We will refer to this architecture, illustrated in
The move from hand-designed features to learned features in machine learning
has been wildly successful. In spite of this, optimization algorithms are still
designed by hand. In this paper we show how the design of an optimization
algorithm can be cast as a learning problem, allowing the algorithm to learn to
exploit structure in the problems of interest in an automatic way. Our learned
algorithms, implemented by LSTMs, outperform generic, hand-designed competitors
on the tasks for which they are trained, and also generalize well to new tasks
with similar structure. We demonstrate this on a number of tasks, including
simple convex problems, training neural networks, and styling images with
neural art.
Marcin Andrychowicz Misha Denil Sergio Gomez Matthew W. Hoffman David Pfau Tom Schaul Brendan Shillingford Nando de Freitas