title: Exploring Data Scaling Trends and Effects in Reinforcement Learning from
    Human Feedback
summary: Reinforcement Learning from Human Feedback (RLHF) is crucial for aligning
  large language models with human preferences. While recent research has focused
  on algorithmic improvements, the importance of prompt-data construction has
  been overlooked. This paper addresses this gap by exploring data-driven
  bottlenecks in RLHF performance scaling, particularly reward hacking and
  decreasing response diversity. We introduce a hybrid reward system combining
  reasoning task verifiers (RTV) and a generative reward model (GenRM) to
  mitigate reward hacking. We also propose a novel prompt-selection method,
  Pre-PPO, to maintain response diversity and enhance learning effectiveness.
  Additionally, we find that prioritizing mathematical and coding tasks early in
  RLHF training significantly improves performance. Experiments across two model
  sizes validate our methods' effectiveness and scalability. Results show that
  RTV is most resistant to reward hacking, followed by GenRM with ground truth,
  and then GenRM with SFT Best-of-N responses. Our strategies enable rapid
  capture of subtle task-specific distinctions, leading to substantial
  improvements in overall RLHF performance. This work highlights the importance
  of careful data construction and provides practical methods to overcome
  performance barriers in RLHF.
paper_authors._id: 67e9fdd446d9dd867e9728d3
paper_authors.name: Wei Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-31T08:24:54.522Z
paper_authors._id: 6468823272d9180d4ac90bdf
paper_authors.avatarUrl: /avatars/70cb7d65d30ecb944595000ceeeedb1b.svg
paper_authors.fullname: Wei Shen
paper_authors.type: user
paper_authors.user: Swtheking
paper_authors._id: 67e9fdd446d9dd867e9728d4
paper_authors.name: Guanlin Liu
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-31T02:28:37.898Z
paper_authors._id: 67805c4a43a58ab7b52a05ea
paper_authors.avatarUrl: /avatars/759d0466020b6f7c0207aaf62ad89eca.svg
paper_authors.fullname: Guanlin Liu
paper_authors.type: user
paper_authors.user: glnbyte
paper_authors._id: 67e9fdd446d9dd867e9728d5
paper_authors.name: Zheng Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-31T08:25:03.076Z
paper_authors._id: 648223754de983d03190f4af
paper_authors.avatarUrl: /avatars/36c70a6a3a1aa8a7cc0de106d5902a81.svg
paper_authors.fullname: Zheng Wu
paper_authors.type: user
paper_authors.user: zhengwu07
paper_authors._id: 67e9fdd446d9dd867e9728d6
paper_authors.name: Ruofei Zhu
paper_authors._id: 67e9fdd446d9dd867e9728d7
paper_authors.name: Qingping Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-31T08:25:15.803Z
paper_authors._id: 64d20e1821aed29b2ffd2d99
paper_authors.avatarUrl: /avatars/b0719319a74e8f51fc8a1404aca367e6.svg
paper_authors.fullname: Qingping Yang
paper_authors.type: user
paper_authors.user: qingping95
paper_authors._id: 67e9fdd446d9dd867e9728d8
paper_authors.name: Chao Xin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-31T08:25:23.410Z
paper_authors._id: 661bca6576ac250a1106bfa6
paper_authors.avatarUrl: /avatars/200327d87103f13f7cbbb40d11f2f188.svg
paper_authors.fullname: Chao Xin
paper_authors.type: user
paper_authors.user: amusingchao
paper_authors._id: 67e9fdd446d9dd867e9728d9
paper_authors.name: Yu Yue
paper_authors._id: 67e9fdd446d9dd867e9728da
paper_authors.name: Lin Yan
paper_title: Exploring Data Scaling Trends and Effects in Reinforcement Learning from
    Human Feedback
paper_summary: Reinforcement Learning from Human Feedback (RLHF) is crucial for aligning
  large language models with human preferences. While recent research has focused
  on algorithmic improvements, the importance of prompt-data construction has
  been overlooked. This paper addresses this gap by exploring data-driven
  bottlenecks in RLHF performance scaling, particularly reward hacking and
  decreasing response diversity. We introduce a hybrid reward system combining
  reasoning task verifiers (RTV) and a generative reward model (GenRM) to
  mitigate reward hacking. We also propose a novel prompt-selection method,
  Pre-PPO, to maintain response diversity and enhance learning effectiveness.
  Additionally, we find that prioritizing mathematical and coding tasks early in
  RLHF training significantly improves performance. Experiments across two model
  sizes validate our methods' effectiveness and scalability. Results show that
  RTV is most resistant to reward hacking, followed by GenRM with ground truth,
  and then GenRM with SFT Best-of-N responses. Our strategies enable rapid
  capture of subtle task-specific distinctions, leading to substantial
  improvements in overall RLHF performance. This work highlights the importance
  of careful data construction and provides practical methods to overcome
  performance barriers in RLHF.
paper_ai_summary: This study enhances RLHF performance by addressing reward hacking and response diversity through a hybrid reward system and a novel prompt-selection method.
paper_ai_keywords: Reinforcement Learning from Human Feedback (RLHF) reasoning task verifiers (RTV) generative reward model (GenRM) Pre-PPO reward hacking response diversity mathematical tasks coding tasks SFT Best-of-N responses