Transductive Off-policy Proximal Policy Optimization
<|reference_start|>Transductive Off-policy Proximal Policy Optimization: Proximal Policy Optimization (PPO) is a popular model-free reinforcement learning algorithm, esteemed for its simplicity and efficacy. However, due to its inherent on-policy nature, its proficiency in harnessing data from disparate policies is constrained. This paper introduces a novel off-policy extension to the original PPO method, christened Transductive Off-policy PPO (ToPPO). Herein, we provide theoretical justification for incorporating off-policy data in PPO training and prudent guidelines for its safe application. Our contribution includes a novel formulation of the policy improvement lower bound for prospective policies derived from off-policy data, accompanied by a computationally efficient mechanism to optimize this bound, underpinned by assurances of monotonic improvement. Comprehensive experimental results across six representative tasks underscore ToPPO's promising performance.<|reference_end|>
@article{gan2024transductive,
    title={Transductive Off-policy Proximal Policy Optimization},
    author={Yaozhong Gan, Renye Yan, Xiaoyang Tan, Zhe Wu, Junliang Xing},
    journal={arXiv preprint arXiv:2406.03894},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.03894},
    primaryClass={cs.LG}
}