DEEP ONLINE LEARNING VIA META-LEARNING: CONTINUAL ADAPTATION FOR MODEL-BASED RL
Humans and animals can learn complex predictive models that allow them to accurately and reliably reason about real-world phenomena, and they can adapt such models extremely quickly in the face of unexpected changes. Deep neural network models allow us to represent very complex functions, but lack this capacity for rapid online adaptation. The goal in this paper is to develop a method for continual online learning from an incoming stream of data, using deep neural network models. We formulate an online learning procedure that uses stochastic gradient descent to update model parameters, and an expectation maximization algorithm with a Chinese restaurant process prior to develop and maintain a mixture of models to handle non-stationary task distributions. This allows for all models to be adapted as necessary, with new models instantiated for task changes and old models recalled when previously seen tasks are encountered again. Furthermore, we observe that meta-learning can be used to meta-train a model such that this direct online adaptation with SGD is effective, which is otherwise not the case for large function approximators. In this work, we apply our meta-learning for online learning (MOLe) approach to model-based reinforcement learning, where adapting the predictive model is critical for control; we demonstrate that MOLe outperforms alternative prior methods, and enables effective continuous adaptation in non-stationary task distributions such as varying terrains, motor failures, and unexpected disturbances.
The paper presents a nonparametric mixture model of neural networks for learning in an environment with a nonstationary distribution. The problem setup includes having access to only a few "modes" of the distribution. Training of the initial model occurs with MAML, and distributional changes during test/operation are handled by a combination of online adaptation and creations of new mixture components when necessary. The mixture is nonparametric and modeled with a CRP. The application considered in the paper is RL, and the experiments compare proposed model against baselines that do not utilize meta-learning (achieved in the proposed method with MAML), and baselines which utilize only a single model component.
I thought the combination of meta-learning and a CRP was a neat way to tackle the problem of modeling and learning the "modes" of a nonstationary distribution. Applications in other domains would have been nice, but the presented results in RL sufficiently demonstrate the benefits of the proposed method.
* Questions/Comments
Figure 3 left vs right?
Is the test in the middle of Algorithm 1 correct?