image (left), style (right) and image generated by the LSTM optimizer (center). Left: The result of
applying the training style at the training resolution to a test image. Right: The result of applying a
new style to a test image at double the resolution on which the optimizer was trained.
We train optimizers using only 1 style and 1800 content images taken from ImageNet [Deng et al.,
2009]. We randomly select 100 content images for testing and 20 content images for validation of
trained optimizers. We train the optimizer on 64x64 content images from ImageNet and one ﬁxed
style image. We then test how well it generalizes to a different style image and higher resolution
(128x128). Each image was optimized for 128 steps and trained optimizers were unrolled for 32
steps. Figure 9 shows the result of styling two different images using the LSTM optimizer. The
LSTM optimizer uses inputs preprocessing described in Appendix A and no postprocessing. See
Appendix C for additional images.
Figure 8 compares the performance of the LSTM optimizer to standard optimization algorithms. The
LSTM optimizer outperforms all standard optimizers if the resolution and style image are the same
as the ones on which it was trained. Moreover, it continues to perform very well when both the
resolution and style are changed at test time.
The move from hand-designed features to learned features in machine learning
has been wildly successful. In spite of this, optimization algorithms are still
designed by hand. In this paper we show how the design of an optimization
algorithm can be cast as a learning problem, allowing the algorithm to learn to
exploit structure in the problems of interest in an automatic way. Our learned
algorithms, implemented by LSTMs, outperform generic, hand-designed competitors
on the tasks for which they are trained, and also generalize well to new tasks
with similar structure. We demonstrate this on a number of tasks, including
simple convex problems, training neural networks, and styling images with
neural art.
Marcin Andrychowicz Misha Denil Sergio Gomez Matthew W. Hoffman David Pfau Tom Schaul Brendan Shillingford Nando de Freitas