Cost-Effective Proxy Reward Model Construction with On-Policy and Active Learning
<|reference_start|>Cost-Effective Proxy Reward Model Construction with On-Policy and Active Learning: Reinforcement learning with human feedback (RLHF), as a widely adopted approach in current large language model pipelines, is \textit{bottlenecked by the size of human preference data}. While traditional methods rely on offline preference dataset constructions, recent approaches have shifted towards online settings, where a learner uses a small amount of labeled seed data and a large pool of unlabeled prompts to iteratively construct new preference data through self-generated responses and high-quality reward/preference feedback. However, most current online algorithms still focus on preference labeling during policy model updating with given feedback oracles, which incurs significant expert query costs. \textit{We are the first to explore cost-effective proxy reward oracles construction strategies for further labeling preferences or rewards with extremely limited labeled data and expert query budgets}. Our approach introduces two key innovations: (1) on-policy query to avoid OOD and imbalance issues in seed data, and (2) active learning to select the most informative data for preference queries. Using these methods, we train a evaluation model with minimal expert-labeled data, which then effectively labels nine times more preference pairs for further RLHF training. For instance, our model using Direct Preference Optimization (DPO) gains around over 1% average improvement on AlpacaEval2, MMLU-5shot and MMLU-0shot, with only 1.7K query cost. Our methodology is orthogonal to other direct expert query-based strategies and therefore might be integrated with them to further reduce query costs.<|reference_end|>
@article{chen2024cost-effective,
    title={Cost-Effective Proxy Reward Model Construction with On-Policy and Active
  Learning},
    author={Yifang Chen, Shuohang Wang, Ziyi Yang, Hiteshi Sharma, Nikos
  Karampatziakis, Donghan Yu, Kevin Jamieson, Simon Shaolei Du, Yelong Shen},
    journal={arXiv preprint arXiv:2407.02119},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.02119},
    primaryClass={cs.LG cs.AI cs.CL}
}