Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization
<|reference_start|>Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization: Reinforcement Learning from Human Feedback (RLHF) has achieved impressive empirical successes while relying on a small amount of human feedback. However, there is limited theoretical justification for this phenomenon. Additionally, most recent studies focus on value-based algorithms despite the recent empirical successes of policy-based algorithms. In this work, we consider an RLHF algorithm based on policy optimization (PO-RLHF). The algorithm is based on the popular Policy Cover-Policy Gradient (PC-PG) algorithm, which assumes knowledge of the reward function. In PO-RLHF, knowledge of the reward function is not assumed, and the algorithm uses trajectory-based comparison feedback to infer the reward function. We provide performance bounds for PO-RLHF with low query complexity, which provides insight into why a small amount of human feedback may be sufficient to achieve good performance with RLHF. A key novelty is a trajectory-level elliptical potential analysis, which bounds the reward estimation error when comparison feedback (rather than numerical reward observation) is given. We provide and analyze algorithms PG-RLHF and NN-PG-RLHF for two settings: linear and neural function approximation, respectively.<|reference_end|>
@article{du2024exploration-driven,
    title={Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on
  Efficient Data Utilization},
    author={Yihan Du, Anna Winnicki, Gal Dalal, Shie Mannor, R. Srikant},
    journal={arXiv preprint arXiv:2402.10342},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.10342},
    primaryClass={cs.LG cs.AI cs.CL}
}