Model-based reinforcement learning has the potential to be more sample efficient than model-free approaches. However, existing model-based methods are vulnerable to model bias, which leads to poor generalization and asymptotic performance compared to model-free counterparts. In this paper, we propose a novel policy optimization framework using an uncertainty-aware objective function to handle those issues. In this framework, the agent simultaneously learns an uncertaintyaware dynamics model and optimizes the policy according to these learned models. Under this framework, the objective function can represented end-to-end as a single computational graph, which allows seamless policy gradient computation via backpropagation through the models. In addition to being theoretically sound, our approach shows promising results on challenging continuous control benchmarks with competitive asymptotic performance and sample complexity compared to state-of-the-art baselines.
# Introduction
The motivation of this paper relies on the dynamics model not being accurate enough, which leads to compounding errors. Hence, a proper characterization of the uncertainty is needed. However, the model-based methods that suffer the most of this problem are the ones that are build on top of policy gradients (since they need to predict the entire trajectory) (e.g., [6]). The methods that learn a Q-value function from the model do not suffer as much from this problem since they just predict shorter horizons. Current model-based RL methods that learn a Q or value-function take into account the uncertainty (i.e., STEVE, MBPO). Those methods are not “less competitive in terms of asymptotic performance.” 
There has been work on learning a parametric policy from MPC. Therefore, you can extract a parametric policy from the optimization that MPC performs. The statement “Not being able to explicitly represent the policy makes it hard to transfer the learned policy to other tasks or to initialize agents with an existing better-than-random policy” is not true. The MPC will transfer better to other tasks that have the same dynamics, since it is not task specific. Given the learned dynamics model and the reward function you can act optimally in any new task as long as the learned dynamics are valid.
# Related work
My main concern with the related work section is that there a lot of literature on risk sensitive and optimism in the face of uncertainty (which is a subset of your method when c>0) in control, bandits, and some on *reinforcement learning* that has been neglected. 
# Uncertainty-Aware Model-Based Policy Optimization
As said before, risk-sensitive in reinforcement learning has been done before and there’s even more work on control and bandits. For instance in [1] (page 5, paragraph (b)) has the same equation and they discuss the effect of the constant being negative or positive. More recent work has also used similar formulations [2].
This section mostly contains previous work, e.g., bootstrap rollout, policy gradient, using a deterministic policy ([3, 4, 5]). One thing that it’s still not clear from reading the paper, are you backpropagating the through the dynamics model, are you using a policy gradient method (REINFORCE, TRPO, PPO,… )?
# Algorithm Summary
One of the novelties introduced is the fact that the data used in each model comes from sampling a Poisson variable. However, this is not ablated in the results sections. Is it necessary? [6] Claims that there’s no need to use different data for the learned models.
# Experiment
The experiment section lacks from more complex environments, in this case the most complex is half-cheetah. Furthermore, given that 3 of the tasks are short horizon tasks you should probably also compare against model-based methods that build on top of policy gradients (e.g., [6]). 
It seems that some choices in the algorithm are not ablated: 1) use of poisson, 2) use of deterministic vs stochastic policy, 3) Is there a single risk that works across environments? Which environments are risk prone/adverse? 4) How about having c ~ N(0, 1), effectively modelling V as a gaussian?
-----------------------------------
Overall, the paper is not mature enough to be accepted: there is not enough novelty, and the results lack of novelty, enough delta in performance from prior work, and have high variance.
------------------------------------
Minor/Typos:
First paragraph: “trying model the transition”
What does it mean that the accuracy is not satisfied?
Why the related work on deep model-based reinforcement learning is called Deep Neural Networks?
3.2 third paragraph: “Next we provide a convergence convergence …”
3.3.2, first paragraph: “no matter how uncertain it may know about the world”
Why the axis in the results section mean different things?
[1] Risk-sensitive Reinforcement Learning. Yun Shen, Michael J. Tobia, Tobias Sommer, Klaus Obermayer. 
[2] Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control. Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, Igor Mordatch
[3] Continuous control with deep reinforcement learning. Timothy P. Lillicrap et. al.
[4] Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning. Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I. Jordan, Joseph E. Gonzalez, Sergey Levine
[5] Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion. Jacob Buckman, Danijar Hafner, George Tucker, Eugene Brevdo, Honglak Lee.
[6] Model-Ensemble Trust-Region Policy Optimization. Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, Pieter Abbeel.