We study the problem of safe adaptation: given a model trained on a variety of past experiences for some task, can this model learn to perform that task in a new situation while avoiding catastrophic failure? This problem setting occurs frequently in real-world reinforcement learning scenarios such as a vehicle adapting to drive in a new city, or a robotic drone adapting a policy trained only in simulation. While learning without catastrophic failures is exceptionally difficult, prior experience can allow us to learn models that make this much easier. These models might not directly transfer to new settings, but can enable cautious adaptation that is substantially safer than naïve adaptation as well as learning from scratch. Building on this intuition, we propose risk-averse domain adaptation (RADA). RADA works in two steps: it first trains probabilistic model-based RL agents in a population of source domains to gain experience and capture epistemic uncertainty about the environment dynamics. Then, when dropped into a new environment, it employs a pessimistic exploration policy, selecting actions that have the best worst-case performance as forecasted by the probabilistic model. We show that this simple maximin policy accelerates domain adaptation in a safety-critical driving environment with varying vehicle sizes. We compare our approach against other approaches for adapting to new environments, including meta-reinforcement learning.
This paper proposes to adapt RL agents from some set of training environments (which, in the current instantiation, vary in some simple respect) to a new domain. They build on a framework for model-based RL called PETS. 
The approach goes as follows: 
2-step process
 * train probabilistic model-based RL agents in a “population of source domains”
 * dropped into new environment use “pessimistic exploration policy”
Then at test time, in order to compute estimates for the rewards for each action the authors use a “particle propagation” technique for unrolling through their dynamics model .
The action is chosen by looking at the sum of the 0 through kth percentile rewards. 
This is a weird choice. Why are they looking at a sum over quantiles vs a quantile itself?
The claim is that the models from the first stage capture the epistemic uncertainty due to not knowing z.
However, the authors give a too scant a treatment of what these uncertainty estimates really mean.
For example, they appear to only be valid with respect to an assumed distribution over z.
The paper’s experiments however focus in large part on what happens when the model is evaluated 
on values of z that were outside the support of the distribution over training domains. 
In this case, any benefit appears to be ill explained by the underlying motivation.
The next step here is to finetune the model as data is collected on the new domain.
Authors propose heuristics for this finetuning that include
1. Drawing experiences from the past experiences (under different domains) and 
2. “keeping the model close to the original model”, via some sort of regularization presumably.
>>> 	why isn’t the exact nature of how they “keep the model near the original model explained in the text?
	perhaps the authors mean that 1. and 2. are one and the same (1 as  means to achieve 2)
	if this is the case, then the exposition should be improved to make this more clear.
Some important details appear to be missing. For example, how many distinct source domains are seen during pretraining? Do they set z different z for every single episode of pretraining? Some language here is unclear, for example what precisely does an “iteration” mean in the context of the experiments? 
The choice to report “average maximum reward” seems strange if what the authors care about is avoiding risk. Can they explain/justify this choice or if not, present a much more comprehensive set of experimental results?
The figures tracking catastrophic failures vs performance resembles those in 
“Combating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear”  https://arxiv.org/abs/1611.01211
This raises some question about why they don’t if concerned with “catastrophic events” model them more explicitly. 
Else, if the return accurately captures all desiderata, why to we need to count the failures?
In short this is a simpzle empirical paper that makes use of heuristic uncertainty estimates, 
including in settings when the estimates have no validity. The writing is reasonably clear
and the ideas are straightforward (which is perfectly fine!). A few of the decisions are unnatural,
a few are ad hoc, and a few details are missing. Overall my sense is that this paper 
has some good qualitities, including the clarity of much of the exposition, 
but it’s still below the mark to be an impactful ICLR paper. 
==========UPDATE=================
I read the rebuttal and am glad that the authors took time to read my review and engage with the criticism as well as try to make some small improvements to the paper, especially exploring the impact on the number of training environments on the results (in the original paper the number of environments available at train time was unlimited). The answers to some of the other questions were less convincing. E.g. the seemingly incoherent objective of summing over the quantiles falls flat. Why should we care more about being a "strict generalization" of some previous algorithm built upon than of having a coherent objective? Overall, I don't think the paper makes it over the bar to accept but I hope the authors continue to improve upon the work and get it into shape where it could be accepted at another strong conference.