three convolutional layers with max pooling followed by a fully-connected layer with 32 hidden units;
all non-linearities were ReLU activations with batch normalization.
The coordinatewise network decomposition introduced in Section 2.1—and used in the previous
experiment—utilizes a single LSTM architecture with shared weights, but separate hidden states,
for each optimizee parameter. We found that this decomposition was not sufﬁcient for the model
architecture introduced in this section due to the differences between the fully connected and convolutional layers. Instead we modify the optimizer by introducing two LSTMs: one proposes parameter
updates for the fully connected layers and the other updates the convolutional layer parameters. Like
the previous LSTM optimizer we still utilize a coordinatewise decomposition with shared weights
and individual hidden states, however LSTM weights are now shared only between parameters of the
same type (i.e. fully-connected vs. convolutional).
The performance of this trained optimizer compared against the baseline techniques is shown in
Figure 7. The left-most plot displays the results of using the optimizer to ﬁt a classiﬁer on a held-out
test set. The additional two plots on the right display the performance of the trained optimizer on
The move from hand-designed features to learned features in machine learning
has been wildly successful. In spite of this, optimization algorithms are still
designed by hand. In this paper we show how the design of an optimization
algorithm can be cast as a learning problem, allowing the algorithm to learn to
exploit structure in the problems of interest in an automatic way. Our learned
algorithms, implemented by LSTMs, outperform generic, hand-designed competitors
on the tasks for which they are trained, and also generalize well to new tasks
with similar structure. We demonstrate this on a number of tasks, including
simple convex problems, training neural networks, and styling images with
neural art.
Marcin Andrychowicz Misha Denil Sergio Gomez Matthew W. Hoffman David Pfau Tom Schaul Brendan Shillingford Nando de Freitas