Learning to Optimize (Li & Malik, 2016) is a recently proposed framework for learning optimization algorithms using reinforcement learning. In this paper, we explore learning an optimization algorithm for training shallow neural nets. Such high-dimensional stochastic optimization problems present interesting challenges for existing reinforcement learning algorithms. We develop an extension that is suited to learning optimization algorithms in this setting and demonstrate that the learned optimization algorithm consistently outperforms other known optimization algorithms even on unseen tasks and is robust to changes in stochasticity of gradients and the neural net architecture. More specifically, we show that an optimization algorithm trained with the proposed method on the problem of training a neural net on MNIST generalizes to the problems of training neural nets on the Toronto Faces Dataset, CIFAR-10 and CIFAR-100.
Summary of the paper
---------------------------
The paper derives a scheme for learning optimization algorithm for high-dimensional stochastic problems as the one involved in shallow neural nets training. The main motivation is to learn to optimize with the goal to design a meta-learner able to generalize across optimization problems (related to machine learning applications as learning a neural network) sharing the same properties. For this sake, the paper casts the problem into reinforcement learning framework and relies on guided policy search (GPS) to explore the space of states and actions. The states are represented by the iterates, the gradients, the objective function values, derived statistics and features, the actions are the update directions of parameters to be learned. To make the formulated problem tractable, some simplifications are introduced (the policies are restricted to gaussian distributions family, block diagonal structure is imposed on the involved parameters). The mean of the stationary non-linear policy of GPS is modeled as a recurrent network with parameters to be learned. A hatch of how to learn the overall process is presented. Finally experimental evaluations on synthetic or real datasets are conducted to show the effectiveness of the approach.
Comments
-------------
- The overall idea of the paper, learning how to optimize, is very seducing and the experimental evaluations (comparison to normal optimizers and other meta-learners) tend to conclude the proposed method is able to learn the behavior of an optimizer and to generalize to unseen problems.
- Materials of the paper sometimes appear tedious to follow, mainly in sub-sections 3.4 and 3.5. It would be desirable to sum up the overall procedure in an algorithm. Page 5, the term $\omega$ intervening in the definition of the policy $\pi$ is not defined.
- The definitions of the statistics and features (state and observation features) look highly elaborated. Can authors provide more intuition on these precise definitions? How do they impact for instance changing the time range in the definition of $\Phi$) in the performance of the meta-learner?
- Figures 3 and 4 illustrate some oscillations of the proposed approach. Which guarantees do we have that the algorithm will not diverge as L2LBGDBGD does? How long should be the training to ensure a good and stable convergence of the method?
- An interesting experience to be conducted and shown is to train the meta-learner on another dataset (CIFAR for example) and to evaluate its generalization ability on the other sets to emphasize the effectiveness of the method.