trained on the neural art task.
Histories of updates We select a single optimizee parameter (one color channel of one pixel in the
styled image) and trace the updates proposed to this coordinate by the LSTM optimizer over a single
trajectory of optimization. We also record the updates that would have been proposed by both SGD
and ADAM if they followed the same trajectory of iterates. Figure 10 shows the trajectory of updates
for two different optimizee parameters. From the plots it is clear that the trained optimizer makes
bigger updates than SGD and ADAM. It is also visible that it uses some kind of momentum, but its
updates are more noisy than those proposed by ADAM which may be interpreted as having a shorter
time-scale momentum.
Proposed update as a function of current gradient Another way to visualize the optimizer
behavior is to look at the proposed update gtfor a single coordinate as a function of the current
gradient evaluation rt. We follow the same procedure as in the previous experiment, and visualize
the proposed updates for a few selected time steps.
These results are shown in Figures 11–13. In these plots, the x-axis is the current value of the gradient
for the chosen coordinate, and the y-axis shows the update that each optimizer would propose should
the corresponding gradient value be observed. The history of gradient observations is the same for all
The move from hand-designed features to learned features in machine learning
has been wildly successful. In spite of this, optimization algorithms are still
designed by hand. In this paper we show how the design of an optimization
algorithm can be cast as a learning problem, allowing the algorithm to learn to
exploit structure in the problems of interest in an automatic way. Our learned
algorithms, implemented by LSTMs, outperform generic, hand-designed competitors
on the tasks for which they are trained, and also generalize well to new tasks
with similar structure. We demonstrate this on a number of tasks, including
simple convex problems, training neural networks, and styling images with
neural art.
Marcin Andrychowicz Misha Denil Sergio Gomez Matthew W. Hoffman David Pfau Tom Schaul Brendan Shillingford Nando de Freitas