We propose a new mixture of model-based and model-free reinforcement learning (RL) algorithms that combines the strengths of both RL methods. Our goal is to reduce the sample complexity of model-free approaches utilizing fictitious trajectory rollouts performed on a learned dynamics model to improve the data efficiency of policy gradient methods while maintaining the same asymptotic behaviour. We suggest to use a special type of uncertainty quantification by a stochastic dynamics model in which the next state prediction is randomly drawn from the distribution predicted by the dynamics model. As a result, the negative effect of exploiting erroneously optimistic regions in the dynamics model is addressed by next state predictions based on an uncertainty aware ensemble of dynamics models. The influence of the ensemble of dynamics models on the policy update is controlled by adjusting the number of virtually performed rollouts in the next iteration according to the ratio of the real and virtual total reward. Our approach, which we call Model-Based Policy Gradient Enrichment (MBPGE), is tested on a collection of benchmark tests including simulated robotic locomotion. We compare our approach to plain model-free algorithms and a model-based one. Our evaluation shows that MBPGE leads to higher learning rates in an early training stage and an improved asymptotic behaviour.
### Summary ###
This paper focuses on model based reinforcement learning (RL). Specifically, the authors consider the setting of combining model based and model free RL algorithms by using the learned dynamics model to generate new data for training the model free algorithm. In order to capture the uncertainty of the environment and the model, the author applied Baysian neural network to learn the dynamics of the environment. The authors approximated the true Bayesian inference process by keeping an anchored ensemble of neural networks, where the prior and posterior of network weights are approximately Gaussian. The ensemble of dynamics model is then used to generate data to train a PPO[1] based agent. In order to prevent the agent from exploiting the learned dynamics model, the authors propose a heuristic way of balancing the amount of real data and model generated data by comparing the rewards.
The authors evaluate the proposed algorithm on simulated robotic locomotion environments in MuJoCo, and the results show that the proposed method has better same efficiency compared to baseline methods in some environments.
### Review ###
Overall I think this paper presents an interesting idea in combing model based and model free RL algorithms. The idea is very well presented and authors include empirical evidence to support the proposed method. However I do find a number of shortcomings that need to be addressed.
Pro:
1. The idea for this paper is really well presented. The structure of the paper is well organized and the experiment results are easy to interpret.
2.  The authors provide a detailed description of the configurations and the hyperparameters for each experiments. Such description would be very helpful if the results in this paper are to be reproduced.
Con:
1. I’m not convinced about the magnitude of novelty in this paper. The proposed method seems very similar to ME-TRPO[2] and it seems to me that the novelty comes from the application of Bayesian ensemble techniques and the generated data ratio tuning heuristics. While these variations might be important for the final performance of the proposed method, the paper does not include any ablation study to further justify the importance of these variations.
2. I’m not convinced about some of the performance of some of the baseline methods presented in this paper. In this paper, MB-MPO[3] does not improve at all during training on Half-Cheetah environment. However, in the original MB-MPO paper, the algorithm does improve and the performance seems to be comparable to that of the proposed method in this paper.
3. The experiment results are not very strong for the proposed method. In 3 of 4 environments, the proposed algorithm does not show much advantage over the baseline algorithms. The only environment in which the proposed method shows significant improvement is Half-Cheetah, and I believe that the baseline algorithms might not be properly tuned in this environment.
4. The paper lacks certain baseline comparisons. There are many other model based RL algorithms developed recently, and it would be important to compare to these methods. Some examples would be ME-TRPO[2], SLBO[4] and MBPO[5].
The idea in the paper is well presented and carefully investigated. However, I am still not convinced about the novelty of the proposed idea and the magnitude of performance improvement. Therefore, I would not recommend acceptance before these problems are addressed. 
References
[1] Schulman, John, et al. "Proximal policy optimization algorithms." arXiv preprint arXiv:1707.06347 (2017).
[2] Kurutach, Thanard, et al. "Model-ensemble trust-region policy optimization." arXiv preprint arXiv:1802.10592 (2018).
[3] Clavera, Ignasi, et al. "Model-based reinforcement learning via meta-policy optimization." arXiv preprint arXiv:1809.05214 (2018).
[4] Luo, Yuping, et al. "Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees." arXiv preprint arXiv:1807.03858 (2018).
[5] Janner, Michael, et al. "When to Trust Your Model: Model-Based Policy Optimization." arXiv preprint arXiv:1906.08253 (2019).