LEARNING TO ADAPT IN DYNAMIC, REAL-WORLD ENVIRONMENTS THROUGH META-REINFORCEMENT LEARNING
Although reinforcement learning methods can achieve impressive results in simulation, the real world presents two major challenges: generating samples is exceedingly expensive, and unexpected perturbations or unseen situations cause proficient but specialized policies to fail at test time. Given that it is impractical to train separate policies to accommodate all situations the agent may see in the real world, this work proposes to learn how to quickly and effectively adapt online to new tasks. To enable sample-efficient learning, we consider learning online adaptation in the context of model-based reinforcement learning. Our approach uses meta-learning to train a dynamics model prior such that, when combined with recent data, this prior can be rapidly adapted to the local context. Our experiments demonstrate online adaptation for continuous control tasks on both simulated and real-world agents. We first show simulated agents adapting their behavior online to novel terrains, crippled body parts, and highly-dynamic environments. We also illustrate the importance of incorporating online adaptation into autonomous agents that operate in the real world by applying our method to a real dynamic legged millirobot. We demonstrate the agent's learned ability to quickly adapt online to a missing leg, adjust to novel terrains and slopes, account for miscalibration or errors in pose estimation, and compensate for pulling payloads.
This work addresses the problem of online adapting dynamics models in the context of model-based RL. Learning globally accurate dynamics model is impossible if we consider that environments are dynamic and we can't observe every possible environment state at initial training time. Thus learning dynamics models that can be adapted online fast, to deal with unexpected und never seen before events is an important research problem.
This paper proposes to use meta-learning to train an update policy that can update the dynamics model at test time in a sample efficient manner. Two methods are proposed
- GrBAL: this method uses MAML for meta-learning
- ReBAL: this method trains a recurrent network during meta-training such that it can update the dynamics effectively at test time when the dynamics  change
Both methods are evaluated on several simulation environments, which show that GrBAL outperforms ReBAL (on average). GrBAL is then evaluated on a real system. 
The strengths of this paper are:
- this work addresses an important problem and is well motivated
- experiments on both simulated and on a real system are performed
The weaknesses:
- the related work section is biased towards the ML community. There is a ton of work on adapting (inverse) dynamics models in the robotics community. This line of work is almost entirely ignored in this paper. Furthermore some important recent references for model-based RL are not provided in the related work section (PETS [3] and MPPI [2]), although MPPI is the controller that is used in this work as a framework for model-based RL. Additionally, existing work on model-based RL with meta-learning [1] has not been cited. This is unacceptable. 
- There is no significant technical contribution - the "contribution" is that existing meta-learning methods have been applied to the model-based RL setting. Even if no-one has had that idea before - it would be a minor contribution, but given that there is prior work on meta-learning in the context of model-based RL, this idea itself is not novel anymore.
- Two methods are provided, without much analysis. Often authors refer to "our approach" - but it's actually not clear what they mean by our approach. The authors can't claim "model-based meta RL" as their approach. 
- While I commend the authors for performing both simulation and real-world experiments, I find the that experiments lack a principled evaluation. More details below.
Feedback on experiments:
Section 6.2 (sample efficiency)
You compare apples to oranges here. I have no idea whether your improvements in terms of sample-efficiency are due to using a model-based RL approach or because your deploying meta-learning. It is well known that model-based RL is more sample efficient, but often cannot achieve the same asymptotic performance as model-free RL. Since MPPI is your choice of model-based RL framework, you would have to include an evaluation that shows results on MPPI with model bootstrapping (as presented in [2]) to give us an idea of how much more sample-efficient your approach is.
Section 6.3 (fast adaptation and generalization)
While in theory one can choose the meta-learning approach independently from the choice of model-based controller, in practice the choice of the MPC method is very important. MPPI can handle model inaccuracies very well - almost to the point where sometimes adaptation is not necessary. You CANNOT evaluate MPPI with online adaptation to another MPC approach with another model-learning approach. This does not give me any information of how your meta-learning improves model-adaptation. In essence these comparisons are meaningless. To make your results more meaningful you need to use the same controller setup (let's say MPPI) and then compare the following:
1. MPPI with your meta-trained online adaptation
2. MPPI results with a fixed learned dynamics model - this shows us whether online adaptation helps
3. results of MPPI with the initial dynamics model (trained in the meta-training phase) -without online adaptation. This will tell us whether the meta-training phase provides a dynamics model that generalizes better (even without online adaptation)
4. MPPI with model bootstrapping (as presented in [2]). This will show whether your meta-trained online adaptation actually outperforms simple online model bootstrapping in terms of sample-efficiency
The key here is that you need to use the same model-based control setup (whether its MPPI or some other method). Otherwise you cannot detangle the effect of controller choice from your meta-learned online adaptation.
6.4 Real-world: same comments as above, comparisons are not meaningful
[1] Meta Reinforcement Learning with Latent Variable Gaussian Processes, UAI 2018
[2] MPPI with model-bootstrapping: Information Theoretic MPC for Model-Based Reinforcement Learning , ICRA 2017
[3] Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models, NIPS 2018