Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning
<|reference_start|>Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning: Constrained Reinforcement Learning (CRL) tackles sequential decision-making problems where agents are required to achieve goals by maximizing the expected return while meeting domain-specific constraints, which are often formulated as expected costs. In this setting, policy-based methods are widely used since they come with several advantages when dealing with continuous-control problems. These methods search in the policy space with an action-based or parameter-based exploration strategy, depending on whether they learn directly the parameters of a stochastic policy or those of a stochastic hyperpolicy. In this paper, we propose a general framework for addressing CRL problems via gradient-based primal-dual algorithms, relying on an alternate ascent/descent scheme with dual-variable regularization. We introduce an exploration-agnostic algorithm, called C-PG, which exhibits global last-iterate convergence guarantees under (weak) gradient domination assumptions, improving and generalizing existing results. Then, we design C-PGAE and C-PGPE, the action-based and the parameter-based versions of C-PG, respectively, and we illustrate how they naturally extend to constraints defined in terms of risk measures over the costs, as it is often requested in safety-critical scenarios. Finally, we numerically validate our algorithms on constrained control problems, and compare them with state-of-the-art baselines, demonstrating their effectiveness.<|reference_end|>
@article{montenegro2024last-iterate,
    title={Last-Iterate Global Convergence of Policy Gradients for Constrained
  Reinforcement Learning},
    author={Alessandro Montenegro and Marco Mussi and Matteo Papini and Alberto
  Maria Metelli},
    journal={arXiv preprint arXiv:2407.10775},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.10775},
    primaryClass={cs.LG}
}