Over the last decade, two competing control strategies have emerged for solving complex control tasks with high efficacy. Model-based control algorithms, such as model-predictive control (MPC) and trajectory optimization, peer into the gradients of underlying system dynamics in order to solve control tasks with high sample efficiency. However, like all gradient-based numerical optimization methods, model-based control methods are sensitive to initialization and are prone to becoming trapped in local minima. Deep reinforcement learning (DRL), on the other hand, can somewhat alleviate these issues by exploring the solution space through sampling -at the expense of computational cost. In this paper, we present a hybrid method that combines the best aspects of gradient-based methods and DRL. We base our algorithm on the deep deterministic policy gradients (DDPG) algorithm and propose a simple modification that uses true gradients from a differentiable physical simulator to increase the convergence rate of both the actor and the critic. We demonstrate our algorithm on seven 2D robot control tasks, with the most complex one being a differentiable half cheetah with hard contact constraints. Empirical results show that our method boosts the performance of DDPG without sacrificing its robustness to local minima.
This paper studies optimal control problems where a physical simulator of the system is available, which outputs the gradient of the dynamics. Using the gradients proposed by the model, the authors propose to add two additional terms in the loss function for critic training in DDPG, where these to terms corresponding to the prediction error of $\nabla_{a} Q(s,a)$ and $\nabla_b Q(s,a)$, respectively. However, my main concern is that the form of gradient given in equation (2) might contains an error.
1. Equation (2). Note that in DDPG, the action is given by a deterministic policy. Thus, we have $a_t = \pi(s_t)$ for all $t\geq 0$. For critic estimation, it seems you are basing on the Bellman equation 
$ Q(s,a) = r(s,a) + Q(s', \pi(s'))$, where $s'$ is the next state following $(s,a)$. Then, it seems that Equation (2) is obtained by taking gradient with respect to $(s,a)$. However, I cannot understand what $\nabla_{\pi} Q$ stands for. If it is $\nabla_a Q(s_{i+1}, a_{i+1}) \cdot \nabla_s \pi(s_{i+1}) $, then that makes sense. 
2. Based on the experiments, it seems that the proposed method does not always outperform MPC or DDPG, even in a small-scale control problem Mountaincar. Moreover, it seems that the performance is similar to that of the DDPG. 
3. Here the model-based gradient in equation (2) is defined by only unroll one-step forward by going from $s_i, a_i$ to $s_{i+1}$. It would be interesting to see how the number of unroll steps affect the algorithm, which is a gradient version of TD($\lambda$).
4. Missing reference: Differential Temporal Difference Learning https://arxiv.org/abs/1812.11137