title: RLAIF: Scaling Reinforcement Learning from Human Feedback with AI
    Feedback
summary: Reinforcement learning from human feedback (RLHF) is effective at aligning
  large language models (LLMs) to human preferences, but gathering high quality
  human preference labels is a key bottleneck. We conduct a head-to-head
  comparison of RLHF vs. RL from AI Feedback (RLAIF) - a technique where
  preferences are labeled by an off-the-shelf LLM in lieu of humans, and we find
  that they result in similar improvements. On the task of summarization, human
  evaluators prefer generations from both RLAIF and RLHF over a baseline
  supervised fine-tuned model in ~70% of cases. Furthermore, when asked to rate
  RLAIF vs. RLHF summaries, humans prefer both at equal rates. These results
  suggest that RLAIF can yield human-level performance, offering a potential
  solution to the scalability limitations of RLHF.
paper_authors._id: 64f529842335edf7651f8230
paper_authors.name: Harrison Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-06T20:55:59.343Z
paper_authors._id: 64f8bbea01b8087098b1d178
paper_authors.avatarUrl: /avatars/2885f343c6e3c27c5116fe9f3912b2b1.svg
paper_authors.fullname: Harrison Lee
paper_authors.type: user
paper_authors.user: harrisonl
paper_authors._id: 64f529842335edf7651f8231
paper_authors.name: Samrat Phatale
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-08T19:32:02.465Z
paper_authors._id: 64fb65f348411fc78979a654
paper_authors.avatarUrl: /avatars/66b61203f2a6fd07f9026ba97f9e8c5d.svg
paper_authors.fullname: Samrat Phatale
paper_authors.type: user
paper_authors.user: samratph
paper_authors._id: 64f529842335edf7651f8232
paper_authors.name: Hassan Mansoor
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-04T15:37:01.907Z
paper_authors._id: 629098fd5463575364e7697a
paper_authors.avatarUrl: /avatars/e45f3ee28bff3571f76caf847d3c36db.svg
paper_authors.fullname: Hassan Mansoor
paper_authors.type: user
paper_authors.user: HassanMansoor
paper_authors._id: 64f529842335edf7651f8233
paper_authors.name: Kellie Lu
paper_authors._id: 64f529842335edf7651f8234
paper_authors.name: Thomas Mesnard
paper_authors._id: 64f529842335edf7651f8235
paper_authors.name: Colton Bishop
paper_authors._id: 64f529842335edf7651f8236
paper_authors.name: Victor Carbune
paper_authors._id: 64f529842335edf7651f8237
paper_authors.name: Abhinav Rastogi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-26T07:55:31.905Z
paper_authors._id: 653966932a101a151da1155c
paper_authors.avatarUrl: /avatars/10baa6536793b397c05028011d4ffcdc.svg
paper_authors.fullname: Abhinav Rastogi
paper_authors.type: user
paper_authors.user: abhirast
paper_title: RLAIF: Scaling Reinforcement Learning from Human Feedback with AI
    Feedback
paper_summary: Reinforcement learning from human feedback (RLHF) is effective at aligning
  large language models (LLMs) to human preferences, but gathering high quality
  human preference labels is a key bottleneck. We conduct a head-to-head
  comparison of RLHF vs. RL from AI Feedback (RLAIF) - a technique where
  preferences are labeled by an off-the-shelf LLM in lieu of humans, and we find
  that they result in similar improvements. On the task of summarization, human
  evaluators prefer generations from both RLAIF and RLHF over a baseline
  supervised fine-tuned model in ~70% of cases. Furthermore, when asked to rate
  RLAIF vs. RLHF summaries, humans prefer both at equal rates. These results
  suggest that RLAIF can yield human-level performance, offering a potential
  solution to the scalability limitations of RLHF.
paper_ai_keywords: Reinforcement learning from human feedback (RLHF) large language models (LLMs) RL from AI Feedback (RLAIF) summarization human evaluators supervised fine-tuned model human-level performance