ALGORITHMIC FRAMEWORK FOR MODEL-BASED DEEP REINFORCEMENT LEARNING WITH THEORETI- CAL GUARANTEES
Model-based reinforcement learning (RL) is considered to be a promising approach to reduce the sample complexity that hinders model-free RL. However, the theoretical understanding of such methods has been rather limited. This paper introduces a novel algorithmic framework for designing and analyzing model-based RL algorithms with theoretical guarantees. We design a meta-algorithm with a theoretical guarantee of monotone improvement to a local maximum of the expected reward. The meta-algorithm iteratively builds a lower bound of the expected reward based on the estimated dynamical model and sample trajectories, and then maximizes the lower bound jointly over the policy and the model. The framework extends the optimism-in-face-of-uncertainty principle to non-linear dynamical models in a way that requires no explicit uncertainty quantification. Instantiating our framework with simplification gives a variant of model-based RL algorithms Stochastic Lower Bounds Optimization (SLBO). Experiments demonstrate that SLBO achieves stateof-the-art performance when only one million or fewer samples are permitted on a range of continuous control benchmark tasks.
The paper proposed a framework to design model-based RL algorithms. The framework is based on OFU and within this framework the authors develop an algorithm (a variant of SLBO) achieving SOTA performance on MuJoCo tasks.
The paper is very well written and the topic is important for the RL community. The authors do a good job at covering related works, the bounds are very interesting and the results quite convincing. 
Questions/comments to the authors:
1) In footnote 3 you state that "[...] we only need to approximate the dynamical model accurately on the trajectories of the optimal policy". Why only of the optimal policy? Don't you also need an accurate dynamic model for the current policy to perform a good policy improvement step? 
2) A major challenge in RL is that the state distribution \rho^\pi changes with \pi and it is usually very hard to estimate. Therefore, many algorithms assume it does not change if the policy is subject to small changes (examples are PPO and TRPO). In Eq 4.3 it seems that you also do something similar, fixing \rho^\pi and constraining the KL of \pi (and not of the joint distribution p(s,a)). Am I correct? Can you elaborate it a bit more, building a connection with other RL methods?
3) In Eq. 6.1 and 6.2 you minimize the H-loss, defined as the prediction error of your model. Recently, Pathak et al. used the same loss function in many papers (such as Curiosity-driven Exploration by Self-supervised Prediction) and your Eq. 6.2 looks like theirs. The practical implementation of your algorithm looks very similar to theirs too. Can you comment on that? 
4) If I understood it correctly, your V-function directly depends on your model, i.e., you have V(M(s)) and you learn the model M parameters to maximize V. This means that you want to learn the model that, together with the policy, maximizes V. Am I correct? Can you comment a bit more on that? Did you try to optimize them (V and M) separately, i.e., to add a third parameter to learn (the V-function parameters)?
5) How does you algorithm deal with environmental noise? The tasks used for the evaluation are all deterministic and I believe that this heavily simplifies the model learning. It would be interesting an evaluation on a simple problem (for example the swing-up pendulum) in the presence of noise on the observations and/or the transition function.
6) I appreciate that you provide many details about the implementation in the appendix. Can you comment a bit more? Which are the most important hyperparameters? The number of policy optimization n_policy or of model optimization n_model? You mention that you observed policy overfitting at the first iterations. Did you also experience model overfitting? Did normalizing the state help a lot?