This paper proposes a differentiable linear quadratic Model Predictive Control (MPC) framework for safe imitation learning. The infinite-horizon cost is enforced using a terminal cost function obtained from the discrete-time algebraic Riccati equation (DARE), so that the learned controller can be proven to be stabilizing in closed-loop. A central contribution is the derivation of the analytical derivative of the solution of the DARE, thereby allowing the use of differentiation-based learning methods. A further contribution is the structure of the MPC optimization problem: an augmented Lagrangian method ensures that the MPC optimization is feasible throughout training whilst enforcing hard constraints on state and input, and a prestabilizing controller ensures that the MPC solution and derivatives are accurate at each iteration. The learning capabilities of the framework are demonstrated in a set of numerical studies.
This paper shows how to make infinite-horizon MPC differentiable
by differentiating through the terminal cost function and controller.
Recent work in non-convex finite-horizon continuous control [1,2,3] face
a huge issue in selecting the controller's horizon length and
better-understanding differentiable infinite horizon
control has potentially strong applications in these domains.
As a step in this non-convex direction, this paper provides a nice
investigation in the convex LTI case.
The imitation learning experiments on a small spring dynamical
system are a necessary sanity check for further work, but
many other more complex systems could be empirically studied
and would have made this paper stronger.
One point that would be useful to clarify: the DARE solution in (7,8) is
derived to optimally control a LTI system *without* control/state bounds but
is then used to control the LTI system *with* control/state bounds in (4).
Does this lead to suboptimal solutions to the true infinite-horizon problem?
[1] Chua, K., Calandra, R., McAllister, R., & Levine, S. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. NeurIPS 2018.
[2] Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., & Davidson, J. Learning latent dynamics for planning from pixels. ICML 2019.
[3] Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, Jimmy Ba. Benchmarking Model-Based Reinforcement Learning. arXiv 2019.