A Primal-Dual Algorithm for Offline Constrained Reinforcement Learning with Linear MDPs
<|reference_start|>A Primal-Dual Algorithm for Offline Constrained Reinforcement Learning with Linear MDPs: We study offline reinforcement learning (RL) with linear MDPs under the infinite-horizon discounted setting which aims to learn a policy that maximizes the expected discounted cumulative reward using a pre-collected dataset. Existing algorithms for this setting either require a uniform data coverage assumptions or are computationally inefficient for finding an $\epsilon$-optimal policy with $O(\epsilon^{-2})$ sample complexity. In this paper, we propose a primal dual algorithm for offline RL with linear MDPs in the infinite-horizon discounted setting. Our algorithm is the first computationally efficient algorithm in this setting that achieves sample complexity of $O(\epsilon^{-2})$ with partial data coverage assumption. Our work is an improvement upon a recent work that requires $O(\epsilon^{-4})$ samples. Moreover, we extend our algorithm to work in the offline constrained RL setting that enforces constraints on additional reward signals.<|reference_end|>
@article{hong2024a,
    title={A Primal-Dual Algorithm for Offline Constrained Reinforcement Learning
  with Linear MDPs},
    author={Kihyuk Hong, Ambuj Tewari},
    journal={arXiv preprint arXiv:2402.04493},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.04493},
    primaryClass={stat.ML cs.LG}
}