for the chosen coordinate, and the y-axis shows the update that each optimizer would propose should
the corresponding gradient value be observed. The history of gradient observations is the same for all
methods and follows the trajectory of the LSTM optimizer.
11
The shape of this function for the LSTM optimizer is often step-like, which is also the case for
ADAM. Surprisingly the step is sometimes in the opposite direction as for ADAM, i.e. the bigger the
gradient, the bigger the update.
C Neural Art
Shown below are additional examples of images styled using the LSTM optimizer. Each triple
consists of the content image (left), style (right) and image generated by the LSTM optimizer (center).
D Information sharing between coordinates
In previous sections we considered a coordinatewise architecture, which corresponds by analogy
to a learned version of RMSprop or ADAM. Although diagonal methods are quite effective in
practice, we can also consider learning more sophisticated optimizers that take the correlations
between coordinates into effect. To this end, we introduce a mechanism allowing different LSTMs to
communicate with each other.
Global averaging cells
The simplest solution is to designate a subset of the cells in each LSTM layer for communication.
These cells operate like normal LSTM cells, but their outgoing activations are averaged at each step
The move from hand-designed features to learned features in machine learning
has been wildly successful. In spite of this, optimization algorithms are still
designed by hand. In this paper we show how the design of an optimization
algorithm can be cast as a learning problem, allowing the algorithm to learn to
exploit structure in the problems of interest in an automatic way. Our learned
algorithms, implemented by LSTMs, outperform generic, hand-designed competitors
on the tasks for which they are trained, and also generalize well to new tasks
with similar structure. We demonstrate this on a number of tasks, including
simple convex problems, training neural networks, and styling images with
neural art.
Marcin Andrychowicz Misha Denil Sergio Gomez Matthew W. Hoffman David Pfau Tom Schaul Brendan Shillingford Nando de Freitas