RLAIF: Scaling Reinforcement Learning from Human Feedback with AI
  Feedback
Reinforcement learning from human feedback (RLHF) is effective at aligning
large language models (LLMs) to human preferences, but gathering high quality
human preference labels is a key bottleneck. We conduct a head-to-head
comparison of RLHF vs. RL from AI Feedback (RLAIF) - a technique where
preferences are labeled by an off-the-shelf LLM in lieu of humans, and we find
that they result in similar improvements. On the task of summarization, human
evaluators prefer generations from both RLAIF and RLHF over a baseline
supervised fine-tuned model in ~70% of cases. Furthermore, when asked to rate
RLAIF vs. RLHF summaries, humans prefer both at equal rates. These results
suggest that RLAIF can yield human-level performance, offering a potential
solution to the scalability limitations of RLHF.
64f529842335edf7651f8230
Harrison Lee
claimed_verified
2023-09-06T20:55:59.343Z
64f8bbea01b8087098b1d178
/avatars/2885f343c6e3c27c5116fe9f3912b2b1.svg
Harrison Lee
user
harrisonl
64f529842335edf7651f8231
Samrat Phatale
claimed_verified
2023-09-08T19:32:02.465Z
64fb65f348411fc78979a654
/avatars/66b61203f2a6fd07f9026ba97f9e8c5d.svg
Samrat Phatale
user
samratph
64f529842335edf7651f8232
Hassan Mansoor
admin_assigned
2023-09-04T15:37:01.907Z
629098fd5463575364e7697a
/avatars/e45f3ee28bff3571f76caf847d3c36db.svg
Hassan Mansoor
user
HassanMansoor
64f529842335edf7651f8233
Kellie Lu
64f529842335edf7651f8234
Thomas Mesnard
64f529842335edf7651f8235
Colton Bishop
64f529842335edf7651f8236
Victor Carbune
64f529842335edf7651f8237
Abhinav Rastogi
claimed_verified
2024-03-26T07:55:31.905Z
653966932a101a151da1155c
/avatars/10baa6536793b397c05028011d4ffcdc.svg
Abhinav Rastogi
user
abhirast
RLAIF: Scaling Reinforcement Learning from Human Feedback with AI
  Feedback
Reinforcement learning from human feedback (RLHF) is effective at aligning
large language models (LLMs) to human preferences, but gathering high quality
human preference labels is a key bottleneck. We conduct a head-to-head
comparison of RLHF vs. RL from AI Feedback (RLAIF) - a technique where
preferences are labeled by an off-the-shelf LLM in lieu of humans, and we find
that they result in similar improvements. On the task of summarization, human
evaluators prefer generations from both RLAIF and RLHF over a baseline
supervised fine-tuned model in ~70% of cases. Furthermore, when asked to rate
RLAIF vs. RLHF summaries, humans prefer both at equal rates. These results
suggest that RLAIF can yield human-level performance, offering a potential
solution to the scalability limitations of RLHF.
Reinforcement learning from AI feedback (RLAIF) achieves similar performance to reinforcement learning from human feedback (RLHF) in aligning large language models with human preferences, offering a scalable alternative.
Reinforcement learning from human feedback RLHF Reinforcement learning from AI Feedback RLAIF large language models LLMs human preferences preference labels head-to-head comparison summarization human evaluators baseline supervised fine-tuned model human-level performance