End-to-End Differentiable Physics for Learning and Control
We present a differentiable physics engine that can be integrated as a module in deep neural networks for end-to-end learning. As a result, structured physics knowledge can be embedded into larger systems, allowing them, for example, to match observations by performing precise simulations, while achieves high sample efficiency. Specifically, in this paper we demonstrate how to perform backpropagation analytically through a physical simulator defined via a linear complementarity problem. Unlike traditional finite difference methods, such gradients can be computed analytically, which allows for greater flexibility of the engine. Through experiments in diverse domains, we highlight the system's ability to learn physical parameters from data, efficiently match and simulate observed visual behavior, and readily enable control via gradient-based planning methods. Code for the engine and experiments is included with the paper.
This paper proposed a way of implementing a differentiable physics engine based on Linear Complementarity Problem (LCP) where analytical gradients can be computed based on the primal-dual interior point method. As a result, it can be integrated as a module in an end-to-end learning pipeline, such as deep neural networks.  Existing popular physical simulators, such as MuJoCo and Bullet, have also cast contact dynamics simulation as an optimization problem and resorted to techniques similar to LCP for solving the motion equations. The main (but limited) contribution of this work is to make analytical gradient computation more efficient with the interior point method.  The problem of having a differentiable simulator is well motivated. It enables gradient-based learning/planning methods to backpropagate through dynamics. The authors illustrated the benefits of such a differentiable physical simulator in three types of experiments: 1) system identification, i.e., inferring the physical parameters from behaviors by backpropagating gradients to these parameters (e.g., mass); 2) using the physical simulator as a neural network layer for future prediction from simple visual observations; and 3) using it as a model to build controllers via iLQR.   The proposed simulator adopted similar optimization techniques as in popular simulators, such as MuJoCo. Thus, it would suffer from the same issues caused by the “soft” contact model as in other LCP-based simulators, where contact is triggered when the distance of two contacting objects is below a threshold. Such approximation would lead to dynamics mismatch between the simulated contact physics and the real contact physics.  However, the writing and analysis of this work can be significantly improved:  A great amount of technical detail has been missing from the paper. Many of the symbols and notations in equations (e.q., Eq (1), (2), (3)) are undefined, and Sec. 3 deserved an expanded description of the LCP formulation as well as the primal-dual interior point method. For the sake of improving understanding, I would suggest the authors move some detailed motion equations to the supplementary material and instead provide more intuitive explanations of the method.  In addition to the technical details, more analysis and comparison would be beneficial for understanding the speed and scalability of the proposed simulator. I would like to understand how fast the simulator can perform forward steps in some benchmarking tasks, compared to popular choices, e.g., MuJoCo, Bullet, and DART. Another question is the scalability of this simulator, such as how many contact points, objects, collision meshes the simulator can afford. The experiments have been conducted in tasks with a small number of articulated primitive shapes. The merit of such a physical simulator might be hindered by the limited size of problems it can simulate efficiently.  Questions: Fig. 6: what is the main cause of the lower asymptotic performance of the model-based method in comparison to Double Q-Learning? Is it due to the approximate models built for the Cartpole and Breakout, or the iLQR solver used by the controller?  In summary, this work introduces a way of implementing differentiable simulator with LCP techniques that allows efficient compute of analytical gradients. The technical novelty is a bit limited, but the experiment shows preliminary successes of such a differentiable physical simulator in several tasks. The major weakness of this paper is the lack of technical clarity and detailed analysis of this simulator, which has caused difficulty in judging the merit of this simulator in comparison to existing ones. As this paper can be significantly benefited from another round of revision, I am leaning towards rejecting this paper for now.