Stabilizing RLHF through Advantage Model and Selective Rehearsal
Large Language Models (LLMs) have revolutionized natural language processing,
yet aligning these models with human values and preferences using RLHF remains
a significant challenge. This challenge is characterized by various
instabilities, such as reward hacking and catastrophic forgetting. In this
technical report, we propose two innovations to stabilize RLHF training: 1)
Advantage Model, which directly models advantage score i.e., extra reward
compared to the expected rewards and regulates score distributions across tasks
to prevent reward hacking. 2) Selective Rehearsal, which mitigates catastrophic
forgetting by strategically selecting data for PPO training and knowledge
rehearsing. Our experimental analysis on public and proprietary datasets
reveals that the proposed methods not only increase stability in RLHF training
but also achieve higher reward scores and win rates.
650a465cf36bb51c503816ec
Baolin Peng
admin_assigned
2023-09-20T11:51:26.520Z
61942296d5c2ba6daa290357
/avatars/594021cc183c4922d48b46f43772a062.svg
Baolin Peng
user
Baolin
650a465cf36bb51c503816ed
Linfeng Song
admin_assigned
2023-09-20T11:51:59.764Z
64c94eddcb2f1bf0e7db5a4d
/avatars/f7e2532d3c85d5e5b5a02c579ea68c3a.svg
Linfeng Song
user
freesunshine0316
650a465cf36bb51c503816ee
Ye Tian
claimed_verified
2024-10-17T11:32:48.510Z
6414061f996b2e426f23df48
/avatars/db0af49c5f017f7c455a076b8b2212b6.svg
Ye Tian
user
yetian
650a465cf36bb51c503816ef
Lifeng Jin
650a465cf36bb51c503816f0
Haitao Mi
claimed_verified
2024-04-22T06:55:58.910Z
65147a1426fbd558dbd08f1b
/avatars/86574ee2d5c22e940be1c4e50be88675.svg
Haitao Mi
user
haitaominlp
650a465cf36bb51c503816f1
Dong Yu
Stabilizing RLHF through Advantage Model and Selective Rehearsal
Large Language Models (LLMs) have revolutionized natural language processing,
yet aligning these models with human values and preferences using RLHF remains
a significant challenge. This challenge is characterized by various
instabilities, such as reward hacking and catastrophic forgetting. In this
technical report, we propose two innovations to stabilize RLHF training: 1)
Advantage Model, which directly models advantage score i.e., extra reward
compared to the expected rewards and regulates score distributions across tasks
to prevent reward hacking. 2) Selective Rehearsal, which mitigates catastrophic
forgetting by strategically selecting data for PPO training and knowledge
rehearsing. Our experimental analysis on public and proprietary datasets
reveals that the proposed methods not only increase stability in RLHF training
but also achieve higher reward scores and win rates.
Two methods, Advantage Model and Selective Rehearsal, are introduced to enhance the stability and performance of RLHF training for Large Language Models.
RLHF Advantage Model reward hacking catastrophic forgetting Selective Rehearsal PPO training knowledge rehearsal