Exploring Data Scaling Trends and Effects in Reinforcement Learning from
  Human Feedback
Reinforcement Learning from Human Feedback (RLHF) is crucial for aligning
large language models with human preferences. While recent research has focused
on algorithmic improvements, the importance of prompt-data construction has
been overlooked. This paper addresses this gap by exploring data-driven
bottlenecks in RLHF performance scaling, particularly reward hacking and
decreasing response diversity. We introduce a hybrid reward system combining
reasoning task verifiers (RTV) and a generative reward model (GenRM) to
mitigate reward hacking. We also propose a novel prompt-selection method,
Pre-PPO, to maintain response diversity and enhance learning effectiveness.
Additionally, we find that prioritizing mathematical and coding tasks early in
RLHF training significantly improves performance. Experiments across two model
sizes validate our methods' effectiveness and scalability. Results show that
RTV is most resistant to reward hacking, followed by GenRM with ground truth,
and then GenRM with SFT Best-of-N responses. Our strategies enable rapid
capture of subtle task-specific distinctions, leading to substantial
improvements in overall RLHF performance. This work highlights the importance
of careful data construction and provides practical methods to overcome
performance barriers in RLHF.
67e9fdd446d9dd867e9728d3
Wei Shen
admin_assigned
2025-03-31T08:24:54.522Z
6468823272d9180d4ac90bdf
/avatars/70cb7d65d30ecb944595000ceeeedb1b.svg
Wei Shen
user
Swtheking
67e9fdd446d9dd867e9728d4
Guanlin Liu
extracted_pending
2025-03-31T02:28:37.898Z
67805c4a43a58ab7b52a05ea
/avatars/759d0466020b6f7c0207aaf62ad89eca.svg
Guanlin Liu
user
glnbyte
67e9fdd446d9dd867e9728d5
Zheng Wu
admin_assigned
2025-03-31T08:25:03.076Z
648223754de983d03190f4af
/avatars/36c70a6a3a1aa8a7cc0de106d5902a81.svg
Zheng Wu
user
zhengwu07
67e9fdd446d9dd867e9728d6
Ruofei Zhu
67e9fdd446d9dd867e9728d7
Qingping Yang
admin_assigned
2025-03-31T08:25:15.803Z
64d20e1821aed29b2ffd2d99
/avatars/b0719319a74e8f51fc8a1404aca367e6.svg
Qingping Yang
user
qingping95
67e9fdd446d9dd867e9728d8
Chao Xin
admin_assigned
2025-03-31T08:25:23.410Z
661bca6576ac250a1106bfa6
/avatars/200327d87103f13f7cbbb40d11f2f188.svg
Chao Xin
user
amusingchao
67e9fdd446d9dd867e9728d9
Yu Yue
67e9fdd446d9dd867e9728da
Lin Yan
Exploring Data Scaling Trends and Effects in Reinforcement Learning from
  Human Feedback
Reinforcement Learning from Human Feedback (RLHF) is crucial for aligning
large language models with human preferences. While recent research has focused
on algorithmic improvements, the importance of prompt-data construction has
been overlooked. This paper addresses this gap by exploring data-driven
bottlenecks in RLHF performance scaling, particularly reward hacking and
decreasing response diversity. We introduce a hybrid reward system combining
reasoning task verifiers (RTV) and a generative reward model (GenRM) to
mitigate reward hacking. We also propose a novel prompt-selection method,
Pre-PPO, to maintain response diversity and enhance learning effectiveness.
Additionally, we find that prioritizing mathematical and coding tasks early in
RLHF training significantly improves performance. Experiments across two model
sizes validate our methods' effectiveness and scalability. Results show that
RTV is most resistant to reward hacking, followed by GenRM with ground truth,
and then GenRM with SFT Best-of-N responses. Our strategies enable rapid
capture of subtle task-specific distinctions, leading to substantial
improvements in overall RLHF performance. This work highlights the importance
of careful data construction and provides practical methods to overcome
performance barriers in RLHF.
This study enhances RLHF performance by addressing reward hacking and response diversity through a hybrid reward system and a novel prompt-selection method.
Reinforcement Learning from Human Feedback (RLHF) reasoning task verifiers (RTV) generative reward model (GenRM) Pre-PPO reward hacking response diversity mathematical tasks coding tasks SFT Best-of-N responses