Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
Model-based reinforcement learning (RL) algorithms can attain excellent sample efficiency, but often lag behind the best model-free algorithms in terms of asymptotic performance. This is especially true with high-capacity parametric function approximators, such as deep networks. In this paper, we study how to bridge this gap, by employing uncertainty-aware dynamics models. We propose a new algorithm called probabilistic ensembles with trajectory sampling (PETS) that combines uncertainty-aware deep network dynamics models with sampling-based uncertainty propagation. Our comparison to state-of-the-art model-based and model-free deep RL algorithms shows that our approach matches the asymptotic performance of model-free algorithms on several challenging benchmark tasks, while requiring significantly fewer samples (e.g., 8 and 125 times fewer samples than Soft Actor Critic and Proximal Policy Optimization respectively on the half-cheetah task).
This paper describes a model-based reinforcement learning approach which is applied on 4 of the continuous control Mujoco tasks.  The approach incorporates uncertainty in the forward dynamics model in two ways: by predicting a Gaussian distribution over future states, rather than a single point, and by training an ensemble of models using different subsets of the agent's experience.  As a controller, the authors use the CEM method to generate action sequences, which are then used to generate state trajectories using the stochastic forward dynamics model.  Reward sums are computed for each of the action-conditional trajectories, and the action corresponding to the highest predicted reward is executed.  This is thus a form of model-predictive control.   In their experiments, the authors show that their method is able to match the performance of SOTA model-free approaches using many fewer environment interactions, i.e. with improved sample complexity, for 3 out of 4 tasks. It also performs better than another recent model-based RL approach which uses a deterministic forward dynamics model and a random shooting methods, rather than CEM, for producing candidate actions to be evaluated by the forward model.   Quality: The paper performs a thorough experimental evaluation on the 4 tasks and compares against several strong baselines, including model-based and model-free methods. The authors also perform ablation studies which disambiguate the effects of different design decisions, such as probabilistic vs. deterministic outputs of the forward model and single model vs. ensemble of models. These actually show that most of the performance gains come from the use of a probabilistic output, rather than the model ensembling, which somewhat weakens the authors' story in the introduction that modeling both aleatoric and epistemic uncertainty (captured by probabilistic models and ensembling, respectively) is useful, although the gains of using the ensembling are clear for 1 out of 4 tasks.   Clarity: The paper is clear and well-written, and results are well-communicated through the figures.   Originality: As the authors point out, the components of their method are not novel (probabilistic neural networks, ensembling, using dynamics models for planning), but this particular combination of components applied to these particular tasks is.   Significance:  It seems to me that the main contribution of this paper is to provide new evidence that with a well-designed experimental setup, model-based RL methods can, indeed, match model-free methods with much better sample complexity.  The components of this setup aren't particularly novel, but the authors have done a good job of making their method work empirically which sets a higher standard for future model-based methods.   One limitation is that the tasks themselves all have low-dimensional state and actions spaces, which allow the authors to use simple and small fully-connected models together with an explicit paramaterization of the output distribution (as a mean and variance over output states). It is unclear whether the method in its current form would be applicable to more challenging tasks involving higher-dimensional state or action spaces (such as images). Using an explicit paramaterization of a high-dimensional Gaussian is harder to train, and it is not clear if the CEM method they use would scale to high-dimensional action spaces. This may require gradient-based optimization of a paramaterized policy.   Overall, I would suggest a weak accept. The contributions of the paper aren't fundamental, but the respectable empirical results help validate the model-based approach to RL, which I believe to be a useful direction overall for adapting RL methods to more practical applications where improved sample complexity is important.