Transferring learned knowledge from one environment to another is an important step towards practical reinforcement learning (RL). In this paper, we investigate the problem of transfer learning across environments with different dynamics while accomplishing the same task in the continuous control domain. We start by illustrating the limitations of policy-centric methods (policy gradient, actorcritic, etc.) when transferring knowledge across environments. We then propose a general model-based value-centric (MVC) framework for continuous RL. MVC learns a dynamics approximator and a value approximator simultaneously in the source domain, and makes decision based on both of them. We evaluate MVC against popular baselines on 5 benchmark control tasks in a training from scratch setting and a transfer learning setting. Our experiments demonstrate MVC achieves comparable performance with the baselines when it is trained from scratch, while it significantly surpasses them when it is used in the transfer setting.
This paper proposes a model-based value-centric (MVC) framework for transfer learning in continuous RL problems, and an algorithm within that framework. The paper attempts to answer two questions: (1) "why are current RL algorithms so inefficient in transfer learning" and (2) "what kind of RL algorithms could be friendly to transfer learning by nature"? I think these are very interesting questions to investigate, and researchers that work on transfer learning could benefit from insights on them. However, I am not yet convinced that this paper answers these questions satisfyingly. It would be great to hear the author's thoughts on my questions below. 
The main insight I take away from the paper is that policy gradient methods are not suitable for transfer learning compared to model-based and value-centric methods for some assumptions (the reward function not changing and the transition dynamics being deterministic). This insight and the experiments in the paper are interesting, but I am unsure if the paper as it is presented now passes the bar for ICLR.
In general the paper has two contributions:
A) analysis of value-centric vs policy-centric methods
B) an algorithm that is more useful for transfer learning.
Regarding A)
The authors argue that policy-centric algorithms are less useful for transfer learning than value-centric methods. 
They first illustrate this with an example in Section 3. Since this is just one example, as a reader I wonder if it would not be possible to construct an example that shows the exact opposite, where value iteration fails but policy gradient doesn't. It feels like there are many assumptions that play into the given example (the reward function not changing; the transition dynamics being deterministic; the choice of using policy gradients and value iteration). 
In addition, the authors provide a theoretical justification in the Appendix (which I have briefly scanned) and the intuition behind it in Section 5. From what I understand, the main problem arises from the policy's output space being a Gaussian distribution, which causes the policy being able to get stuck in a local optimum. Further, the authors show (in the Appendix) that under some assumtions the value function always converges. Are there any guarantees on this when we don't have access to the true reward and transition functions (which themselves could get stuck in a local optimum)?
Would the authors say that the phenomenon is more a problem with the algorithm (policy gradient vs value iteration) than policy-centric and value-centric methods in general? Are there other methods that would be able to transfer policies better than policy gradient methods?
Regarding B)
The author's proposed method (MVC) has three components: the value function, the dynamics model and the reward model, all of which are learned by neural networks. It seems like the main advantage comes from using a model (since that's the aspect which changes when having to transfer to an altered MDP). Does the advantage of this method over DDPG and TRPO come from the fact that the dynamics model changes smoothly, and we have an approximation to it? Then it is not surprising that this outperforms a policy gradient method. 
Other comments:
- Could you explain what is meant by "precise" and "imprecise" when speaking about policies or value functions?
- Could you explain what is meant by the algorithm being "accessible" (e.g., Definition 1)?
- Section 2.1: In Property 1, what is f? Could you make explicit why we are interested in the two properties listed? By "not rigorously", do you mean that those properties are based on intuition? These properties are used later in the paper and the appendix, so I wonder how strong of an assumption this is.
- Section 2.2: Could you explain what is meant by "task"? You say that within the MDP, the transition dynamics and reward functions change, but the task stays the same. However, earlier (in the introduction) you state that only the environment dynamics change. I find it confusing that "the task" is something hand-wavy and not part of the formal definition of the MDP. In what exact ways can the reward function be influenced by the change in the transition dynamics? 
- Section 3: Replace "obviously" with "hence"; remove "it is not hard to find that". This might not be so trivial for some readers.
- Appendix B: Refer to Table 1 in the text.
Clarity: The paper is written well, but I think some assumptions and their affects should be stated more clearly and put into context. The paper misses a discussion / conclusion section. It would be great to see a discussion on some of the assumptions; e.g., what if the low dimensional assumtion breaks down? What if we assume that also the reward function can change? The authors are in a unique position to give insight into these things (even if the results from the paper do not hold after dropping some assumptions) and it would be very helpful to share these with the reader in a discussion section.