Learning to learn by gradient descent by gradient descent
The move from hand-designed features to learned features in machine learning has been wildly successful. In spite of this, optimization algorithms are still designed by hand. In this paper we show how the design of an optimization algorithm can be cast as a learning problem, allowing the algorithm to learn to exploit structure in the problems of interest in an automatic way. Our learned algorithms, implemented by LSTMs, outperform generic, hand-designed competitors on the tasks for which they are trained, and also generalize well to new tasks with similar structure. We demonstrate this on a number of tasks, including simple convex problems, training neural networks, and styling images with neural art.
Paper 1982: Learning to learn by gradient descent by gradient descent  An LSTM learns entire (gradient-based) learning algorithms for certain classes of functions, extending similar work of the 1990s and early 2000s.
Comments (mostly on related work):   Authors: "The idea of using learning to learn or meta-learning to acquire knowledge or inductive biases has a long history [Thrun and Pratt, 1998]."  But the intro to this reference is muddling the waters by confusing meta-learning (which is about learning the learning algorithm itself) and transfer learning, subsuming basically everything under "meta-learning," even standard back-propagation, because it can be applied to some data set, and then may learn new data points more quickly (so this is just standard transfer learning).    To my knowledge, the first work on learning general learning algorithms written in a  universal programming language was published in 1987:   J.  Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: The meta-meta-... hook. Diploma thesis, Institut für Informatik, Technische Universität München, 1987.  Authors: "This work was built on by [Younger et al., 2001, Hochreiter et al., 2001] wherein a higher-level network act as a gradient descent procedure, with both levels trained during learning. Earlier work of Runarsson and Jonsson [2000] trains similar feed-forward meta-learning rules using evolutionary strategies. Alternatively Schmidhuber [1992, 1993] considers networks that are able to modify their own behavior and act as an alternative to recurrent networks in meta-learning. Note, however that these earlier works do not directly address the transfer of a learned training procedure to novel problem instances and instead focus on adaptivity in the online setting."  This does not quite do justice to the previous work. Schmidhuber's 1993 meta-RNN was not an alternative to RNNs but an RNN able to run arbitrary computable weight change algorithms on the RNN itself. It could sequentially address any of its own weights and read or modify it (a bit like an NTM whose storage cells are actually fast weights). The whole system was differentiable end-to-end such that gradient descent could be used to search for a learning algorithm and also a meta-learning algorithm and a meta-meta-learning algorithm and so on - no restrictions. And Hochreiter's meta-LSTM really did "directly address the transfer of a learned training procedure to novel problem instances" (the words of the authors of the submission) in practice, because it was trained on certain quadratic functions to learn a learning algorithm for quadratic functions, and then a new quadratic function was presented in form of input/target pairs to the input units of the meta-LSTM, which generalized from the previous problem instances (which were functions) to the new ones, such that it could learn the new quadratic function 30 times faster than backpropagation or gradient descent, that is, it really learned by gradient descent to learn much faster than by gradient descent. Whether the resulting learning algorithm used something that's closely related to some fast version of gradient descent (or something more sophisticated) was not analyzed, to my knowledge. But the setup was general enough to permit all kinds of computable learning algorithms.  I think it is very important that the authors make very clear what's different to their own approach, provided there is a major difference at all.   I think the main difference is that the authors explicitly bias their meta-learning setup towards the learning of gradient-based learning algorithms (while the previous work did not have an explicit bias of this kind). This may be a good thing, at least in the context of the studied experimental tasks, and should be emphasized.  Authors: "Finally, Daniel et al. [2016] considers using reinforcement learning to train a controller for selecting step-sizes, however this work is much more constrained than ours and still requires hand-tuned features."   However, there was a much earlier much more general reinforcement learning system that learned to learn learning algorithms (written in a general programming language - no restrictions), namely the success-story algorithm SSA (called EIRA in an earlier  ICML publication):  M. Wiering and J. Schmidhuber. Solving POMDPs using Levin search and EIRA. In L. Saitta, ed., Machine Learning: Proceedings of the 13th International Conference (ICML 1996), pages 534-542, Morgan Kaufmann Publishers, San Francisco, CA, 1996. Based on:  J. Schmidhuber. On learning how to learn learning strategies. TR FKI-198-94, TUM, 1994.  J. Schmidhuber, J. Zhao, and M. Wiering. Shifting inductive bias with success-story algorithm, adaptive Levin search, and incremental self-improvement. Machine Learning 28:105-130, 1997.  This system could run arbitrary algorithms, including arbitrary learning algorithms and meta-learning algorithms affecting its own code, using SSA to favor "good" learning algorithms that optimize reward intake per time interval. It should be easy for the authors of the submission to identify the main differences to their own approach.   Authors: "We implement the update rule for each coordinate using a two-layer Long Short Term Memory (LSTM) network" ...   Did they use the original LSTM of 1997, or the LSTM with forget gates for the recurrent units which most people use?:  F. A. Gers and J. Schmidhuber and F. Cummins. Learning to Forget: Continual Prediction with LSTM. Neural Computation, 12(10):2451--2471, 2000.  "an NTM-BFGS optimizer, because its use of external memory is similar to the Neural Turing Machine [Graves et al., 2014]."  So it is like the fast weight system of 1992 which separates storage and control?  Authors: "Quadratic functions. In this experiment we consider training an optimizer on a simple class of synthetic 10-dimensional quadratic functions."  So this is like the experiments with quadratic functions of Hochreiter et al's 2001 paper. Did the authors perform a direct comparison to the 2001 system?   Conclusion:  Authors: "We have shown how to cast the design of optimization algorithms as a learning problem, which enables us to train optimizers that are specialized to particular classes of functions."   Well, this was already shown by the previous work up to 2001 mentioned above. I think the conclusion should focus on what's really new in this paper.   It seems to me that the novelty lies in the explicit bias towards gradient-based learning algorithms (while the previous work did not have an explicit bias of this kind). This is probably a good bias for the experiments presented in this paper, and should be emphasized.   And the additional experiments (besides those with quadratic functions) are cool, and the results are intriguing! I hope that this paper will help to rekindle interest in gradient-based learning to learn. It should be published, provided the comments above are addressed. I'd like to see the revised version again.