title: Stabilizing RLHF through Advantage Model and Selective Rehearsal
summary: Large Language Models (LLMs) have revolutionized natural language processing,
  yet aligning these models with human values and preferences using RLHF remains
  a significant challenge. This challenge is characterized by various
  instabilities, such as reward hacking and catastrophic forgetting. In this
  technical report, we propose two innovations to stabilize RLHF training: 1)
  Advantage Model, which directly models advantage score i.e., extra reward
  compared to the expected rewards and regulates score distributions across tasks
  to prevent reward hacking. 2) Selective Rehearsal, which mitigates catastrophic
  forgetting by strategically selecting data for PPO training and knowledge
  rehearsing. Our experimental analysis on public and proprietary datasets
  reveals that the proposed methods not only increase stability in RLHF training
  but also achieve higher reward scores and win rates.
paper_authors._id: 650a465cf36bb51c503816ec
paper_authors.name: Baolin Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:51:26.520Z
paper_authors._id: 61942296d5c2ba6daa290357
paper_authors.avatarUrl: /avatars/594021cc183c4922d48b46f43772a062.svg
paper_authors.fullname: Baolin Peng
paper_authors.type: user
paper_authors.user: Baolin
paper_authors._id: 650a465cf36bb51c503816ed
paper_authors.name: Linfeng Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T11:51:59.764Z
paper_authors._id: 64c94eddcb2f1bf0e7db5a4d
paper_authors.avatarUrl: /avatars/f7e2532d3c85d5e5b5a02c579ea68c3a.svg
paper_authors.fullname: Linfeng Song
paper_authors.type: user
paper_authors.user: freesunshine0316
paper_authors._id: 650a465cf36bb51c503816ee
paper_authors.name: Ye Tian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-17T11:32:48.510Z
paper_authors._id: 6414061f996b2e426f23df48
paper_authors.avatarUrl: /avatars/db0af49c5f017f7c455a076b8b2212b6.svg
paper_authors.fullname: Ye Tian
paper_authors.type: user
paper_authors.user: yetian
paper_authors._id: 650a465cf36bb51c503816ef
paper_authors.name: Lifeng Jin
paper_authors._id: 650a465cf36bb51c503816f0
paper_authors.name: Haitao Mi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-22T06:55:58.910Z
paper_authors._id: 65147a1426fbd558dbd08f1b
paper_authors.avatarUrl: /avatars/86574ee2d5c22e940be1c4e50be88675.svg
paper_authors.fullname: Haitao Mi
paper_authors.type: user
paper_authors.user: haitaominlp
paper_authors._id: 650a465cf36bb51c503816f1
paper_authors.name: Dong Yu
paper_title: Stabilizing RLHF through Advantage Model and Selective Rehearsal
paper_summary: Large Language Models (LLMs) have revolutionized natural language processing,
  yet aligning these models with human values and preferences using RLHF remains
  a significant challenge. This challenge is characterized by various
  instabilities, such as reward hacking and catastrophic forgetting. In this
  technical report, we propose two innovations to stabilize RLHF training: 1)
  Advantage Model, which directly models advantage score i.e., extra reward
  compared to the expected rewards and regulates score distributions across tasks
  to prevent reward hacking. 2) Selective Rehearsal, which mitigates catastrophic
  forgetting by strategically selecting data for PPO training and knowledge
  rehearsing. Our experimental analysis on public and proprietary datasets
  reveals that the proposed methods not only increase stability in RLHF training
  but also achieve higher reward scores and win rates.
paper_ai_summary: Two methods, Advantage Model and Selective Rehearsal, are introduced to enhance the stability and performance of RLHF training for Large Language Models.
paper_ai_keywords: RLHF Advantage Model reward hacking catastrophic forgetting Selective Rehearsal PPO training knowledge rehearsal