as the ones on which it was trained. Moreover, it continues to perform very well when both the
resolution and style are changed at test time.
Finally, in Appendix B we qualitatively examine the behavior of the step directions generated by the
learned optimizer.
4 Conclusion
We have shown how to cast the design of optimization algorithms as a learning problem, which
enables us to train optimizers that are specialized to particular classes of functions. Our experiments
have conﬁrmed that learned neural optimizers compare favorably against state-of-the-art optimization
methods used in deep learning. We witnessed a remarkable degree of transfer, with for example the
LSTM optimizer trained on 12,288 parameter neural art tasks being able to generalize to tasks with
49,152 parameters, different styles, and different content images all at the same time. We observed
similar impressive results when transferring to different architectures in the MNIST task.
The results on the CIFAR image labeling task show that the LSTM optimizers outperform handengineered optimizers when transferring to datasets drawn from the same data distribution.
References
F. Bach, R. Jenatton, J. Mairal, and G. Obozinski. Optimization with sparsity-inducing penalties. Foundations
and Trends in Machine Learning , 4(1):1–106, 2012.
8
The move from hand-designed features to learned features in machine learning
has been wildly successful. In spite of this, optimization algorithms are still
designed by hand. In this paper we show how the design of an optimization
algorithm can be cast as a learning problem, allowing the algorithm to learn to
exploit structure in the problems of interest in an automatic way. Our learned
algorithms, implemented by LSTMs, outperform generic, hand-designed competitors
on the tasks for which they are trained, and also generalize well to new tasks
with similar structure. We demonstrate this on a number of tasks, including
simple convex problems, training neural networks, and styling images with
neural art.
Marcin Andrychowicz Misha Denil Sergio Gomez Matthew W. Hoffman David Pfau Tom Schaul Brendan Shillingford Nando de Freitas