people. arXiv Report 1604.00289, 2016.
T. Maley. neuron, 2011. URL https://www.flickr.com/photos/taylortotz101/6280077898 . Creative
Commons Attribution 2.0 Generic.
J. Martens and R. Grosse. Optimizing neural networks with Kronecker-factored approximate curvature. In
International Conference on Machine Learning , pages 2408–2417, 2015.
G. L. Nemhauser and L. A. Wolsey. Integer and combinatorial optimization . John Wiley & Sons, 1988.
Y . Nesterov. A method of solving a convex programming problem with convergence rate o (1/k2). In Soviet
Mathematics Doklady , volume 27, pages 372–376, 1983.
J. Nocedal and S. Wright. Numerical optimization . Springer Science & Business Media, 2006.
M. Riedmiller and H. Braun. A direct adaptive method for faster backpropagation learning: The RPROP
algorithm. In International Conference on Neural Networks , pages 586–591, 1993.
T. P. Runarsson and M. T. Jonsson. Evolution and design of distributed learning rules. In IEEE Symposium on
The move from hand-designed features to learned features in machine learning
has been wildly successful. In spite of this, optimization algorithms are still
designed by hand. In this paper we show how the design of an optimization
algorithm can be cast as a learning problem, allowing the algorithm to learn to
exploit structure in the problems of interest in an automatic way. Our learned
algorithms, implemented by LSTMs, outperform generic, hand-designed competitors
on the tasks for which they are trained, and also generalize well to new tasks
with similar structure. We demonstrate this on a number of tasks, including
simple convex problems, training neural networks, and styling images with
neural art.
Marcin Andrychowicz Misha Denil Sergio Gomez Matthew W. Hoffman David Pfau Tom Schaul Brendan Shillingford Nando de Freitas