title: Policy Filtration in RLHF to Fine-Tune LLM for Code Generation
summary: Reinforcement learning from human feedback (RLHF) is one of the key
  techniques that helps large language models (LLMs) to follow instructions and
  provide helpful and harmless responses. While direct policy optimization
  methods exist, state-of-the-art LLMs adopt RL-based methods (usually PPO) in
  RLHF to train the policy to generate good responses guided by a reward model
  learned from preference data. The main challenge of these methods is the
  inaccuracy of the intermediate reward model, especially in code generation
  tasks that require long and complex reasoning to score a response. We find that
  the reliability of the reward model varies across responses assigned with
  different rewards. This motivates us to filter the samples whose rewards may be
  unreliable to improve signal-to-noise ratio during policy learning, resulting
  in Policy Filtration for Proximal Policy Optimization (PF-PPO). To choose a
  proper policy filtration strategy for a given reward model, the coefficient of
  determination (R^2) between rewards and actual scores on filtered samples
  serves as a good metrics and helps us find several promising strategies. We
  provide extensive experiments to validate the effectiveness of PF-PPO in code
  generation tasks, and find that some variants of PF-PPO are highly effective
  and achieve new state-of-the-art performance across 7-billion-parameter models
  on HumanEval, MBPP, and a new and more challenging LeetCode Contest benchmark.
paper_authors._id: 66e984604e8e48237d978b46
paper_authors.name: Wei Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-17T15:16:03.982Z
paper_authors._id: 6468823272d9180d4ac90bdf
paper_authors.avatarUrl: /avatars/70cb7d65d30ecb944595000ceeeedb1b.svg
paper_authors.fullname: Wei Shen
paper_authors.type: user
paper_authors.user: Swtheking
paper_authors._id: 66e984604e8e48237d978b47
paper_authors.name: Chuheng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-17T15:16:16.744Z
paper_authors._id: 64c32727c370f29a10334d35
paper_authors.avatarUrl: /avatars/a3284bdd2e51f437433b89a96a904448.svg
paper_authors.fullname: ZhangChuheng
paper_authors.type: user
paper_authors.user: zhangchuheng123
paper_title: Policy Filtration in RLHF to Fine-Tune LLM for Code Generation
paper_summary: Reinforcement learning from human feedback (RLHF) is one of the key
  techniques that helps large language models (LLMs) to follow instructions and
  provide helpful and harmless responses. While direct policy optimization
  methods exist, state-of-the-art LLMs adopt RL-based methods (usually PPO) in
  RLHF to train the policy to generate good responses guided by a reward model
  learned from preference data. The main challenge of these methods is the
  inaccuracy of the intermediate reward model, especially in code generation
  tasks that require long and complex reasoning to score a response. We find that
  the reliability of the reward model varies across responses assigned with
  different rewards. This motivates us to filter the samples whose rewards may be
  unreliable to improve signal-to-noise ratio during policy learning, resulting
  in Policy Filtration for Proximal Policy Optimization (PF-PPO). To choose a
  proper policy filtration strategy for a given reward model, the coefficient of
  determination (R^2) between rewards and actual scores on filtered samples
  serves as a good metrics and helps us find several promising strategies. We
  provide extensive experiments to validate the effectiveness of PF-PPO in code
  generation tasks, and find that some variants of PF-PPO are highly effective
  and achieve new state-of-the-art performance across 7-billion-parameter models
  on HumanEval, MBPP, and a new and more challenging LeetCode Contest benchmark.
paper_ai_summary: Policy Filtration for Proximal Policy Optimization (PF-PPO) improves RLHF by filtering unreliable reward samples, enhancing performance in code generation tasks compared to state-of-the-art methods.
paper_ai_keywords: Reinforcement learning from human feedback (RLHF) large language models (LLMs) direct policy optimization RL-based methods PPO reward model policy filtration coefficient of determination ($R^2$) HumanEval MBPP LeetCode Contest benchmark