title: A Long Way to Go: Investigating Length Correlations in RLHF
summary: Great successes have been reported using Reinforcement Learning from Human
  Feedback (RLHF) to align large language models. Open-source preference datasets
  and reward models have enabled wider experimentation beyond generic chat
  settings, particularly to make systems more "helpful" for tasks like web
  question answering, summarization, and multi-turn dialogue. When optimizing for
  helpfulness, RLHF has been consistently observed to drive models to produce
  longer outputs. This paper demonstrates that optimizing for response length is
  a significant factor behind RLHF's reported improvements in these settings.
  First, we study the relationship between reward and length for reward models
  trained on three open-source preference datasets for helpfulness. Here, length
  correlates strongly with reward, and improvements in reward score are driven in
  large part by shifting the distribution over output lengths. We then explore
  interventions during both RL and reward model learning to see if we can achieve
  the same downstream improvements as RLHF without increasing length. While our
  interventions mitigate length increases, they aren't uniformly effective across
  settings. Furthermore, we find that even running RLHF with a reward based
  solely on length can reproduce most of the downstream improvements over the
  initial policy model, showing that reward models in these settings have a long
  way to go.
paper_authors._id: 651f68af5ff73c3cec578fb4
paper_authors.name: Prasann Singhal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-10T08:29:56.571Z
paper_authors._id: 613cb3e1c7a43c281cd417a2
paper_authors.avatarUrl: /avatars/69123ba49c2aa1cd9f3cc5746f4839dc.svg
paper_authors.fullname: Prasann Singhal
paper_authors.type: user
paper_authors.user: PrasannSinghal
paper_authors._id: 651f68af5ff73c3cec578fb5
paper_authors.name: Tanya Goyal
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-10-06T01:53:53.436Z
paper_authors._id: 650b7db9a27ba14323e74421
paper_authors.avatarUrl: /avatars/668a881a1bb394a2115fd8f34fe4e875.svg
paper_authors.fullname: Tanya Goyal
paper_authors.type: user
paper_authors.user: tanyagoyal-p
paper_authors._id: 651f68af5ff73c3cec578fb6
paper_authors.name: Jiacheng Xu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-10-06T02:25:52.755Z
paper_authors._id: 631983d5cb116eab31df5821
paper_authors.avatarUrl: /avatars/6a42c842a9439241ead2ace1d79fc32c.svg
paper_authors.fullname: Jiacheng Xu
paper_authors.type: user
paper_authors.user: jcxu
paper_authors._id: 651f68af5ff73c3cec578fb7
paper_authors.name: Greg Durrett
paper_title: A Long Way to Go: Investigating Length Correlations in RLHF
paper_summary: Great successes have been reported using Reinforcement Learning from Human
  Feedback (RLHF) to align large language models. Open-source preference datasets
  and reward models have enabled wider experimentation beyond generic chat
  settings, particularly to make systems more "helpful" for tasks like web
  question answering, summarization, and multi-turn dialogue. When optimizing for
  helpfulness, RLHF has been consistently observed to drive models to produce
  longer outputs. This paper demonstrates that optimizing for response length is
  a significant factor behind RLHF's reported improvements in these settings.
  First, we study the relationship between reward and length for reward models
  trained on three open-source preference datasets for helpfulness. Here, length
  correlates strongly with reward, and improvements in reward score are driven in
  large part by shifting the distribution over output lengths. We then explore
  interventions during both RL and reward model learning to see if we can achieve
  the same downstream improvements as RLHF without increasing length. While our
  interventions mitigate length increases, they aren't uniformly effective across
  settings. Furthermore, we find that even running RLHF with a reward based
  solely on length can reproduce most of the downstream improvements over the
  initial policy model, showing that reward models in these settings have a long
  way to go.
paper_ai_summary: Optimizing for response length significantly contributes to the improvements observed in Reinforcement Learning from Human Feedback (RLHF) when aligning large language models for helpfulness tasks.
paper_ai_keywords: Reinforcement Learning from Human Feedback (RLHF) reward models preference datasets helpfulness multi-turn dialogue web question answering summarization response length downstream improvements rl