MODEL-ENSEMBLE TRUST-REGION POLICY OPTI- MIZATION
Model-free reinforcement learning (RL) methods are succeeding in a growing number of tasks, aided by recent advances in deep learning. However, they tend to suffer from high sample complexity which hinders their use in real-world domains. Alternatively, model-based reinforcement learning promises to reduce sample complexity, but tends to require careful tuning and, to date, it has succeeded mainly in restrictive domains where simple models are sufficient for learning. In this paper, we analyze the behavior of vanilla model-based reinforcement learning methods when deep neural networks are used to learn both the model and the policy, and we show that the learned policy tends to exploit regions where insufficient data is available for the model to be learned, causing instability in training. To overcome this issue, we propose to use an ensemble of models to maintain the model uncertainty and regularize the learning process. We further show that the use of likelihood ratio derivatives yields much more stable learning than backpropagation through time. Altogether, our approach Model-Ensemble Trust-Region Policy Optimization (ME-TRPO) significantly reduces the sample complexity compared to model-free deep RL methods on challenging continuous control benchmark tasks 1 .
Summary:
The paper proposes to use ensembles of models to overcome a typical problem when training on a learned model: That the policy learns to take advantage of errors of the model.
The models use the same training data but are differentiated by a differente parameter initialization and by training on differently drawn minibatches.
To train the policy, at each step the next state is taken from an uniformly randomly drawn model.
For validation the policy is evaluated on all models and training is stopped early if it doesn't improve on enough of them. 
While the idea to use an ensemble of deep neural networks to estimate their uncertainty is not new, I haven't seen it yet in this context. They successfully show in their experiments that typical levels of performance can be achieved using much less samples from the real environment.
The reduction in required samples is over an order of magnitude for simple environments (Mujoco Swimmer). However, (as expected for model based algorithms) both the performance as well as the reduction in sample complexity gets worse with increasing complexity of the environment. It can still successfully tackle the Humanoid Mujoco task but my guess is that that is close to the upper limit of this algorithm?
Overall the paper is a solid and useful contribution to the field.
*Quality:*
The paper is clearly shows the advantage of the proposed method in the experimental section where it compares to several baselines (and not only one, thank you for that!). 
Things which in my opinion aren't absolutely required in the paper but I would find interesting and useful (e.g. in the appendix) are:
1. How does the runtime (e.g. number of total samples drawn from both the models and the real environment, including for validation purpuses) compare?
From the experiments I would guess that MB-TRPO is about two to three orders of magnitude slower, but having this information would be useful.
2. For more complex environments it seems that training is becoming less stable and performance degradates, especially for the Humanoid environment. A plot like in figure 4 (different number of models) for the humanoid environment could be interesting? Additionally maybe a short discussion where the major problem for further scaling lies? For example: Expressiveness of the models? Required number of models / computation feasibility? Etc... This is not necessarily required for the paper but would be interesting.
*Originality & Significance:*
As far as I can tell, none of the fundamental ideas are new. However, they are combined in an interesting, novel way that shows significant performance improvements.
The problem the authors tackle, namely learning a deep neural network model for model based RL, is important and relevant. As such, the paper contributes to the field and should be accepted.
*Smaller questions and notes:*
- Longer training times for MB-TRPO, in particular for Ant and Humanoid would have been intersting if computationionally feasibly.
- Could this in principle be used with Q-learning as well (instead of TRPO) if the action space is discrete? Or is there an obvious reason why not that I am missing?