200 400 600 800 1000
Step100CIFAR-5
200 400 600 800 100010-1CIFAR-2
ADAM
RMSprop
SGD
NAG
LSTM
LSTM-subFigure 7: Optimization performance on the CIFAR-10 dataset and subsets. Shown on the left is the
LSTM optimizer versus various baselines trained on CIFAR-10 and tested on a held-out test set. The
two plots on the right are the performance of these optimizers on subsets of the CIFAR labels. The
additional optimizer LSTM-sub has been trained only on the heldout labels and is hence transferring
to a completely novel dataset.
test-set problems are similar enough to those in the training set we see even better generalization than
the baseline optimizers.
3.3 Training a convolutional network on CIFAR-10
Next we test the performance of the trained neural optimizers on optimizing classiﬁcation performance
for the CIFAR-10 dataset [Krizhevsky, 2009]. In these experiments we used a model with both
convolutional and feed-forward layers. In particular, the model used for these experiments includes
three convolutional layers with max pooling followed by a fully-connected layer with 32 hidden units;
The move from hand-designed features to learned features in machine learning
has been wildly successful. In spite of this, optimization algorithms are still
designed by hand. In this paper we show how the design of an optimization
algorithm can be cast as a learning problem, allowing the algorithm to learn to
exploit structure in the problems of interest in an automatic way. Our learned
algorithms, implemented by LSTMs, outperform generic, hand-designed competitors
on the tasks for which they are trained, and also generalize well to new tasks
with similar structure. We demonstrate this on a number of tasks, including
simple convex problems, training neural networks, and styling images with
neural art.
Marcin Andrychowicz Misha Denil Sergio Gomez Matthew W. Hoffman David Pfau Tom Schaul Brendan Shillingford Nando de Freitas