Bilevel reinforcement learning via the development of hyper-gradient without lower-level convexity
<|reference_start|>Bilevel reinforcement learning via the development of hyper-gradient without lower-level convexity: Bilevel reinforcement learning (RL), which features intertwined two-level problems, has attracted growing interest recently. The inherent non-convexity of the lower-level RL problem is, however, to be an impediment to developing bilevel optimization methods. By employing the fixed point equation associated with the regularized RL, we characterize the hyper-gradient via fully first-order information, thus circumventing the assumption of lower-level convexity. This, remarkably, distinguishes our development of hyper-gradient from the general AID-based bilevel frameworks since we take advantage of the specific structure of RL problems. Moreover, we propose both model-based and model-free bilevel reinforcement learning algorithms, facilitated by access to the fully first-order hyper-gradient. Both algorithms are provable to enjoy the convergence rate $\mathcal{O}(\epsilon^{-1})$. To the best of our knowledge, this is the first time that AID-based bilevel RL gets rid of additional assumptions on the lower-level problem. In addition, numerical experiments demonstrate that the hyper-gradient indeed serves as an integration of exploitation and exploration.<|reference_end|>
@article{yang2024bilevel,
    title={Bilevel reinforcement learning via the development of hyper-gradient
  without lower-level convexity},
    author={Yan Yang, Bin Gao, Ya-xiang Yuan},
    journal={arXiv preprint arXiv:2405.19697},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.19697},
    primaryClass={math.OC cs.AI cs.LG stat.ML}
}