rt=rf(t). This formulation is equivalent to (2)whenwt=1[t=T], but later we will describe
why using different weights can prove useful.
We can minimize the value of L()using gradient descent on . The gradient estimate @L()=@ can
be computed by sampling a random function fand applying backpropagation to the computational
graph in Figure 2. We allow gradients to ﬂow along the solid edges in the graph, but gradients
along the dashed edges are dropped. Ignoring gradients along the dashed edges amounts to making
the assumption that the gradients of the optimizee do not depend on the optimizer parameters, i.e.
@rt
@= 0. This assumption allows us to avoid computing second derivatives of f.
Examining the objective in (3)we see that the gradient is non-zero only for terms where wt6= 0. If
we usewt=1[t=T]to match the original problem, then gradients of trajectory preﬁxes are zero
and only the ﬁnal optimization step provides information for training the optimizer. This renders
Backpropagation Through Time (BPTT) inefﬁcient. We solve this problem by relaxing the objective
The move from hand-designed features to learned features in machine learning
has been wildly successful. In spite of this, optimization algorithms are still
designed by hand. In this paper we show how the design of an optimization
algorithm can be cast as a learning problem, allowing the algorithm to learn to
exploit structure in the problems of interest in an automatic way. Our learned
algorithms, implemented by LSTMs, outperform generic, hand-designed competitors
on the tasks for which they are trained, and also generalize well to new tasks
with similar structure. We demonstrate this on a number of tasks, including
simple convex problems, training neural networks, and styling images with
neural art.
Marcin Andrychowicz Misha Denil Sergio Gomez Matthew W. Hoffman David Pfau Tom Schaul Brendan Shillingford Nando de Freitas