Learning convex bounds for linear quadratic control policy synthesis
Learning to make decisions from observed data in dynamic environments remains a problem of fundamental importance in a number of fields, from artificial intelligence and robotics, to medicine and finance. This paper concerns the problem of learning control policies for unknown linear dynamical systems so as to maximize a quadratic reward function. We present a method to optimize the expected value of the reward over the posterior distribution of the unknown system parameters, given data. The algorithm involves sequential convex programing, and enjoys reliable local convergence and robust stability guarantees. Numerical simulations and stabilization of a real-world inverted pendulum are used to demonstrate the approach, with strong performance and robustness properties observed in both.
Summary: The paper presents a model-based approach for controlling linear systems subject to quadratic costs with stability guarantees. Robust LQ control, yielding a controller K that stabilizes all dynamics (A, B) within an uncertainty set, has previously been studied by Dean et al.  This paper attempts to improve robust solutions (make them less conservative) by optimizing over an approximate dynamics posterior, rather than considering the worst case dynamics. The approach is as follows: - Obtain a set of dynamics samples from estimation posterior and restrict it to stabilizable samples. - Formulate the problem as an SDP in the inverse value matrix X^{-1} and controller K, where the objective is a conservative upper bound on the average control cost of all dynamics samples (A_i, B_i, Pi_i). This is the common Lyapunov (CL) solution. - Propose a less conservative convex upper bound as follows. Given an initial controller \bar{K}, find the dynamics sample with the corresponding lowest cost \bar{X}. Then solve a modified problem where each sample gets its own value matrix X_i but K is still shared, and each X_i^{-1} is approximated using a Taylor series around \bar{X}. This procedure can be applied iteratively to improve CL or other solutions. - The final K stabilizes all dynamics in the convex hull of the samples for the CL approach, and also for the relaxed (proposed) approach if an additional stability constraint is introduced (but I don’t think it is). - Experiments are run on small-scale synthetic dynamics similar to previous work. Data is obtained by exciting the system with Gaussian inputs and force-resetting to 0 every 6 time steps. The proposed algorithm (and variations) do seem to find a good tradeoff between stability and cost; while they occasionally yield unstable controllers, their cost is lower than that of the fully robust controllers. Experiments are also run on inverted pendulum with similar conclusions.  Quality & clarity. This is a high-quality paper with clear writing, explanations, and derivations.  It proposes an interesting approach to robust control of LQ systems with unknown dynamics which trades occasional instability for lower control costs.   Originality: The contributions are mostly extensions of previous work, though the new upper bound is original.   Wrt the SDP formulation: at the beginning of Section 4, authors state that the principal contribution (btw principle-->principal lines 138, 145) of their paper is an SDP formulation of the problem. However, SDP formulations of LQ control have been known for a while, and the paper should include at least one reference.  e.g.  - V. Balakrishnan and L. Vandenberghe. Semidefinite programming duality and linear time-invariant systems. IEEE Transactions on Automatic Control, 2003.  - A Cohen et al. Online linear quadratic control with adversarially changing costs, ICML 2018. (most recently) Standard SDP formulations just solve for X and subsequently obtain K= −(R + B’XB)^{−1}B’XA. The formulation in the paper is only different in that it also optimizes over K in order to tie multiple X_i using the same K.   Significance: While the results are nice (improve overly conservative solutions while mostly maintaining stability), this approach may not be too useful for practical problems due to increased computation reqiurements. Solving LQ control using SDP relaxations is practical only for small-scale systems. In this paper, the computation cost is amplified by the number of dynamics samples and the use of an iterative procedure. This is perhaps not an issue in this paper since the controllers are computed offline from force-reset episodes. But it limits the use of the developed technique in large-scale problems, as well as safety- and stability-critical adaptive settings.   A few additional questions about experiments: - What happens if you make your approach fully stable - is the cost substantially higher? - The nominal controller seems to find lowest-cost solutions (when stable). Why is it not included in the pendulum experiments?