Current model-based reinforcement learning approaches use the model simply as a learned black-box simulator to augment the data for policy optimization or value function learning. In this paper, we show how to make more effective use of the model by exploiting its differentiability. We construct a policy optimization algorithm that uses the pathwise derivative of the learned model and policy across future timesteps. Instabilities of learning across many timesteps are prevented by using a terminal value function, learning the policy in an actor-critic fashion. Furthermore, we present a derivation on the monotonic improvement of our objective in terms of the gradient error in the model and value function. We show that our approach (i) is consistently more sample efficient than existing state-of-the-art model-based algorithms, (ii) matches the asymptotic performance of model-free algorithms, and (iii) scales to long horizons, a regime where typically past model-based approaches have struggled.
#rebuttal responses
Thanks for the clarification!  However, I will keep my original score for these reasons:
(1) Only 3 random seeds are used for each environment, which is not convincing as the variance of MAAC is large in some figures.
(2) Baselines are only trained with 10^5 steps and do not converge. Thus it not fair to say that MAAC matches the asymptotic performance of model-free algorithms.
#review
This paper constructs a model-based policy optimization algorithm (MAAC) that uses the pathwise derivative of the learned model and policy across future timesteps. The terminal value function is used to improve stability.  The theoretical guarantee of the error of the model-based gradient is presented. Experimental results show that MAAC outperforms SAC, STEVE, and MBPO on four environments in terms of the sample efficiency.
The experimental results are strong and I appreciate the plots of the gradient error in a simple task, shown in Figure 3. But I want to see the comparison of the final performance of each algorithm in these environments, and I doubt that the baselines do not converge.
However, the paper is badly written. First of all, the authors claim that the pathwise derivate method is applied to optimize the objective function. But the detail of the method is missing. Secondly, I can not follow the procedure of how Q function is learned in the MAAC algorithm. Thirdly, Theorem 4.1 gives a performance improvement bound of the new policy w.r.t J_pi without the entropy term. But the MAAC algorithm optimizes the objective with the policy entropy term.
Also, MAAC applies many techniques in other papers. The paper does not clearly show the advantage of each component:
(1) I want to see the experimental results of MAAC optimizing the objective without the entropy.  
(2) The SVG(1) algorithm should be compared as a baseline as the SVG(1) also uses the gradient of the learned model to optimize the policy.
(3) The policy and the Q function are optimized on both the real samples and the generated samples. I want to see the ablation study or justification on whether training on real samples or both samples.
Questions:
1. How many independent runs are used in experiments?
2. Does the computation of the pathwise derivate method cost much time？ Is MAAC much slower than SAC?
I am happy to increase my score if the authors justify these questions.