demonstrate this on a number of tasks, including simple convex problems, training
neural networks, and styling images with neural art.
1 Introduction
Frequently, tasks in machine learning can be expressed as the problem of optimizing an objective
functionf()deﬁned over some domain 2. The goal in this case is to ﬁnd the minimizer
= arg min2f(). While any method capable of minimizing this objective function can be
applied, the standard approach for differentiable functions is some form of gradient descent, resulting
in a sequence of updates
t+1=ttrf(t):
The performance of vanilla gradient descent, however, is hampered by the fact that it only makes use
of gradients and ignores second-order information. Classical optimization techniques correct this
behavior by rescaling the gradient step using curvature information, typically via the Hessian matrix
of second-order partial derivatives—although other choices such as the generalized Gauss-Newton
matrix or Fisher information matrix are possible.
Much of the modern work in optimization is based around designing update rules tailored to speciﬁc
classes of problems, with the types of problems of interest differing between different research
The move from hand-designed features to learned features in machine learning
has been wildly successful. In spite of this, optimization algorithms are still
designed by hand. In this paper we show how the design of an optimization
algorithm can be cast as a learning problem, allowing the algorithm to learn to
exploit structure in the problems of interest in an automatic way. Our learned
algorithms, implemented by LSTMs, outperform generic, hand-designed competitors
on the tasks for which they are trained, and also generalize well to new tasks
with similar structure. We demonstrate this on a number of tasks, including
simple convex problems, training neural networks, and styling images with
neural art.
Marcin Andrychowicz Misha Denil Sergio Gomez Matthew W. Hoffman David Pfau Tom Schaul Brendan Shillingford Nando de Freitas