title: RLCD: Reinforcement Learning from Contrast Distillation for Language
    Model Alignment
summary: We propose Reinforcement Learning from Contrast Distillation (RLCD), a method
  for aligning language models to follow natural language principles without
  using human feedback. RLCD trains a preference model using simulated preference
  pairs that contain both a high-quality and low-quality example, generated using
  contrasting positive and negative prompts. The preference model is then used to
  improve a base unaligned language model via reinforcement learning.
  Empirically, RLCD outperforms RLAIF (Bai et al., 2022b) and context
  distillation (Huang et al., 2022) baselines across three diverse alignment
  tasks--harmlessness, helpfulness, and story outline generation--and on both 7B
  and 30B model scales for preference data simulation.
paper_authors._id: 64bf4334af05eb17c71c318e
paper_authors.name: Kevin Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-26T08:44:01.730Z
paper_authors._id: 645bf09b438d6cfbe1b0a73b
paper_authors.avatarUrl: /avatars/6a9165fdaab493d692538005aeb63469.svg
paper_authors.fullname: Kevin Yang
paper_authors.type: user
paper_authors.user: yangkevin2
paper_authors._id: 64bf4334af05eb17c71c318f
paper_authors.name: Dan Klein
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-25T09:00:43.344Z
paper_authors._id: 632be88b3690fb57e70e0bf1
paper_authors.avatarUrl: /avatars/74ff8f30b3662db2602495bdf493d397.svg
paper_authors.fullname: Dan Klein
paper_authors.type: user
paper_authors.user: danjklein
paper_authors._id: 64bf4334af05eb17c71c3190
paper_authors.name: Asli Celikyilmaz
paper_authors._id: 64bf4334af05eb17c71c3191
paper_authors.name: Nanyun Peng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-26T16:15:17.996Z
paper_authors._id: 64c13de54cbd12e168f09a58
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c13de54cbd12e168f09a58/Aw5BaNzf6uuwqQ333OQ5y.png
paper_authors.fullname: NANYUN (Violet) PENG
paper_authors.type: user
paper_authors.user: violetpeng
paper_authors._id: 64bf4334af05eb17c71c3192
paper_authors.name: Yuandong Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-25T09:01:30.739Z
paper_authors._id: 6344cf73ee1504dbcd5bdfe7
paper_authors.avatarUrl: /avatars/6dd2bf1f9c5679e5c8c85d62c9836aac.svg
paper_authors.fullname: Yuandong Tian
paper_authors.type: user
paper_authors.user: tydsh
paper_title: RLCD: Reinforcement Learning from Contrast Distillation for Language
    Model Alignment
paper_summary: We propose Reinforcement Learning from Contrast Distillation (RLCD), a method
  for aligning language models to follow natural language principles without
  using human feedback. RLCD trains a preference model using simulated preference
  pairs that contain both a high-quality and low-quality example, generated using
  contrasting positive and negative prompts. The preference model is then used to
  improve a base unaligned language model via reinforcement learning.
  Empirically, RLCD outperforms RLAIF (Bai et al., 2022b) and context
  distillation (Huang et al., 2022) baselines across three diverse alignment
  tasks--harmlessness, helpfulness, and story outline generation--and on both 7B
  and 30B model scales for preference data simulation.
paper_ai_summary: Reinforcement Learning from Contrast Distillation (RLCD) aligns language models to natural language principles using simulated preference pairs without human feedback, outperforming existing methods across various alignment tasks.
paper_ai_keywords: Reinforcement Learning from Contrast Distillation (RLCD) preference model contrasting positive and negative prompts reinforcement learning RLAIF context distillation alignment tasks harmlessness helpfulness story outline generation