Reinforcement learning algorithms can acquire policies for complex tasks automatically, however the number of samples required to learn a diverse set of skills can be prohibitively large. While meta-reinforcement learning has enabled agents to leverage prior experience to adapt quickly to new tasks, the performance of these methods depends crucially on how close the new task is to the previously experienced tasks. Current approaches are either not able to extrapolate well, or can do so at the expense of requiring extremely large amounts of data due to on-policy training. In this work, we present model identification and experience relabeling (MIER), a meta-reinforcement learning algorithm that is both efficient and extrapolates well when faced with out-of-distribution tasks at test time based on a simple insight: we recognize that dynamics models can be adapted efficiently and consistently with off-policy data, even if policies and value functions cannot. These dynamics models can then be used to continue training policies for out-ofdistribution tasks without using meta-reinforcement learning at all, by generating synthetic experience for the new task.
###  Summary
1. The paper proposes an algorithm capable of off-policy meta-training (Similar to PEARL) as well as off-policy policy adaptation (By relabelling previous data using the adapted model and reward function). 
2. The basic idea is to meta-learn a model that can adapt to different MDPs using a small amount of data. Moreover, the adaptation is done by only changing the latent context vector (Similar to CAVIA or CAML). The remaining parameters of the model (theta) are fixed after meta-training. 
3. The paper also proposes learning a universal policy that, when given the context vector of a task, can maximize the reward for that task. This means that for with-in distribution meta-testing tasks, the policy can be used as it is (by giving it the right context vector which can be computed by adapting the model). For out-of-distribution tasks, however, it is important to update this policy. 
4. To update the policy, the paper proposes combining previously stored data (for example data used in meta-training) with the adapted model to do off-policy learning (Using SAC).  
### Decision with reasons 
I vote for rejecting the paper in its current form for the following reasons:
1- The paper assumes that it is possible to learn models for out-of-distribution tasks with a few samples that are accurate on all the previously stored data. This is fundamentally incorrect. If the MDP changes in a significant way, it is not reasonable to expect that we can adapt a model from a few samples. Moreover, even if we can adapt the model using a lot of new experience, it is not reasonable to expect that we can use this model to accurately label all previous data. The authors do acknowledge this when describing results in Figure 3, however they seem to underplay this limitation. 
2- Turning the meta-RL problem into a supervised learning problem has already been explored. For instance, Nagabandi et al. (2018)[1] showed that it is possible to quickly adapt models to changes using meta-learning. They, however, used decision time planning for the control policy (By random shooting method). This paper, on the other hand, uses Dyna style planning with an off-policy learning algorithm on previously stored data. The only difference is the choice of off-the-shelf planning algorithm which is not a significant contribution (There are some other small differences, such as learning a context vector and not model initialization, learning a universal policy etc, however, I don't see how they are essential for the proposed approach; maybe the authors can clarify why those choices are essential) 
3- The paper assumes a context vector alone is sufficient to capture changes in MDPs (It keeps the rest of the model fixed at adaptation). This might be reasonable if the context vector is sufficiently large, but the paper does not even mention the size of the context vector. It also skips other important details. For example, it does not mention any details about hyper-parameter selection, how the context-vector used in the model, etc. It's hard to judge the importance of the experimental results because of this. 
### Questions 
1- "Effective model training requires the validation batch to contain data corresponding to optimal behavior for the tasks, which we obtain by training a universal policy conditioned on the context descriptor"
It's not clear to me why the validation batch must contain data corresponding to the optimal behavior. 
2- Is the proposed framework really consistent? At adaptation, only the context vector is being updated whereas model parameters (theta) are fixed. Why is a context vector alone sufficient to adapt the model to drastic changes in the MDP? 
[1] https://arxiv.org/abs/1812.07671
### UPDATE
The authors gave a detailed response to the reviews and answered some of my main concerns. However, I'm still not convinced that the paper, in its current form, can be accepted. My issues are: 
The paper combines some existing ideas in a new way but falls short of justifying the choices it made. The proposed contribution is that it is consistent (meta-learning methods that learn a network initialization are also consistent), can do off-line meta-training (So can PEARL) and can use old meta-training data at meta-test time (This is novel to this paper). However,  the proposed methodology also has some downfalls. For example: 
It does not allow continual adaptation. This is an important limitation of existing consistent meta-learning methods and this paper does not address it. Nagabandi et al 18 [1], on the other hand, propose a similar solution that is also capable of continual adaptation. 
MOST IMPORTANTLY, the empirical evaluation in the paper is very unsatisfactory. Even though the authors have included hyper-parameters in the appendix in the updated version of the paper, they still do no specify how these parameters were selected. Were the parameters selected to maximize the performance of their method and then copied for the baselines? This would not be a fair comparison. 
Given the above-mentioned issues, I don't think the paper in its current form can be accepted and I'm maintaining my initial score. I think the authors should do a more thorough empirical investigation and tune the baselines and their method separately (using comparable compute budget). They should also report results on multiple environments using the same parameters (i.e. tune hyper-parameters on one or a few environments and reports results on some other environments as commonly done in Atari) 
[1] Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL