Reinforcement Learning from Human Feedback with Active Queries
<|reference_start|>Reinforcement Learning from Human Feedback with Active Queries: Aligning large language models (LLM) with human preference plays a key role in building modern generative models and can be achieved by reinforcement learning from human feedback (RLHF). Despite their superior performance, current RLHF approaches often require a large amount of human-labelled preference data, which is expensive to collect. In this paper, inspired by the success of active learning, we address this problem by proposing query-efficient RLHF methods. We first formalize the alignment problem as a contextual dueling bandit problem and design an active-query-based proximal policy optimization (APPO) algorithm with an $\tilde{O}(d^2/\Delta)$ regret bound and an $\tilde{O}(d^2/\Delta^2)$ query complexity, where $d$ is the dimension of feature space and $\Delta$ is the sub-optimality gap over all the contexts. We then propose ADPO, a practical version of our algorithm based on direct preference optimization (DPO) and apply it to fine-tuning LLMs. Our experiments show that ADPO, while only making about half of queries for human preference, matches the performance of the state-of-the-art DPO method.<|reference_end|>
@article{ji2024reinforcement,
    title={Reinforcement Learning from Human Feedback with Active Queries},
    author={Kaixuan Ji and Jiafan He and Quanquan Gu},
    journal={arXiv preprint arXiv:2402.09401},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.09401},
    primaryClass={cs.LG cs.AI cs.CL math.OC stat.ML}
}