Policy Gradient Methods for the Cost-Constrained LQR: Strong Duality and Global Convergence
<|reference_start|>Policy Gradient Methods for the Cost-Constrained LQR: Strong Duality and Global Convergence: In safety-critical applications, reinforcement learning (RL) needs to consider safety constraints. However, theoretical understandings of constrained RL for continuous control are largely absent. As a case study, this paper presents a cost-constrained LQR formulation, where a number of LQR costs with user-defined penalty matrices are subject to constraints. To solve it, we propose a policy gradient primal-dual method to find an optimal state feedback gain. Despite the non-convexity of the cost-constrained LQR problem, we provide a constructive proof for strong duality and a geometric interpretation of an optimal multiplier set. By proving that the concave dual function is Lipschitz smooth, we further provide convergence guarantees for the PG primal-dual method. Finally, we perform simulations to validate our theoretical findings.<|reference_end|>
@article{zhao2024policy,
    title={Policy Gradient Methods for the Cost-Constrained LQR: Strong Duality and
  Global Convergence},
    author={Feiran Zhao, Keyou You},
    journal={arXiv preprint arXiv:2406.03734},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.03734},
    primaryClass={math.OC cs.SY eess.SY}
}