title: ODIN: Disentangled Reward Mitigates Hacking in RLHF
summary: In this work, we study the issue of reward hacking on the response length, a
  challenge emerging in Reinforcement Learning from Human Feedback (RLHF) on
  LLMs. A well-formatted, verbose but less helpful response from the LLMs can
  often deceive LLMs or even human evaluators to achieve high scores. The same
  issue also holds for some reward models in RL. To address the challenges in
  both training and evaluation, we establish a more reliable evaluation protocol
  for comparing different training configurations, which inspects the trade-off
  between LLM evaluation score and response length obtained by varying training
  hyperparameters. Based on this evaluation, we conduct large-scale studies,
  where the results shed insights into the efficacy of hyperparameters and tricks
  used in RL on mitigating length bias. We further propose to improve the reward
  model by jointly training two linear heads on shared feature representations to
  predict the rewards, one trained to correlate with length, and the other
  trained to decorrelate with length and therefore focus more on the actual
  content. We then discard the length head in RL to prevent reward hacking on
  length. Experiments demonstrate that our approach almost eliminates the reward
  correlation with length, and improves the obtained policy by a significant
  margin.
paper_authors._id: 65caf98b103cd029421f34e4
paper_authors.name: Lichang Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T11:32:27.148Z
paper_authors._id: 62323bb408bcea92917e42ee
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62323bb408bcea92917e42ee/2vHxkv-oSROtLteOnqa8P.jpeg
paper_authors.fullname: Lichang Chen
paper_authors.type: user
paper_authors.user: Lichang-Chen
paper_authors._id: 65caf98b103cd029421f34e5
paper_authors.name: Chen Zhu
paper_authors._id: 65caf98b103cd029421f34e6
paper_authors.name: Davit Soselia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-13T16:43:33.028Z
paper_authors._id: 64ce047bd1fda042b5238a23
paper_authors.avatarUrl: /avatars/6a98d3cbba23945723762729343e2918.svg
paper_authors.fullname: Davit Soselia
paper_authors.type: user
paper_authors.user: davits
paper_authors._id: 65caf98b103cd029421f34e7
paper_authors.name: Jiuhai Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T11:33:23.455Z
paper_authors._id: 6393847e3e30234ae798b7be
paper_authors.avatarUrl: /avatars/daeb8c37dff4432d837a69b87c196521.svg
paper_authors.fullname: JiuhaiChen
paper_authors.type: user
paper_authors.user: jiuhai
paper_authors._id: 65caf98b103cd029421f34e8
paper_authors.name: Tianyi Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T11:33:48.758Z
paper_authors._id: 647f5af5b0e96764589f3b2a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
paper_authors.fullname: Tianyi Zhou
paper_authors.type: user
paper_authors.user: zhoutianyi
paper_authors._id: 65caf98b103cd029421f34e9
paper_authors.name: Tom Goldstein
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T11:33:59.181Z
paper_authors._id: 6381ca7d65dc156aba0b933d
paper_authors.avatarUrl: /avatars/84dfdca8e1cd6fbf50d6fb2a6f1b488d.svg
paper_authors.fullname: Tom Goldstein
paper_authors.type: user
paper_authors.user: tomgoldstein
paper_authors._id: 65caf98b103cd029421f34ea
paper_authors.name: Heng Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T11:34:06.691Z
paper_authors._id: 6527bc6b34bf5ece73da426d
paper_authors.avatarUrl: /avatars/120739a9ac84e7319d9ea157a63dc547.svg
paper_authors.fullname: henghuang
paper_authors.type: user
paper_authors.user: henghuang
paper_authors._id: 65caf98b103cd029421f34eb
paper_authors.name: Mohammad Shoeybi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-28T07:19:51.117Z
paper_authors._id: 6641544c695975af2cbd0da6
paper_authors.avatarUrl: /avatars/0ad3c18dcba585259b064fe9b00a07ce.svg
paper_authors.fullname: Mohammad Shoeybi
paper_authors.type: user
paper_authors.user: shoeybi
paper_authors._id: 65caf98b103cd029421f34ec
paper_authors.name: Bryan Catanzaro
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T11:34:24.916Z
paper_authors._id: 6311021788942700629e6247
paper_authors.avatarUrl: /avatars/e7adc1632b76e80e7e4a590033d1c20a.svg
paper_authors.fullname: Bryan Catanzaro
paper_authors.type: user
paper_authors.user: ctnzr
paper_title: ODIN: Disentangled Reward Mitigates Hacking in RLHF
paper_summary: In this work, we study the issue of reward hacking on the response length, a
  challenge emerging in Reinforcement Learning from Human Feedback (RLHF) on
  LLMs. A well-formatted, verbose but less helpful response from the LLMs can
  often deceive LLMs or even human evaluators to achieve high scores. The same
  issue also holds for some reward models in RL. To address the challenges in
  both training and evaluation, we establish a more reliable evaluation protocol
  for comparing different training configurations, which inspects the trade-off
  between LLM evaluation score and response length obtained by varying training
  hyperparameters. Based on this evaluation, we conduct large-scale studies,
  where the results shed insights into the efficacy of hyperparameters and tricks
  used in RL on mitigating length bias. We further propose to improve the reward
  model by jointly training two linear heads on shared feature representations to
  predict the rewards, one trained to correlate with length, and the other
  trained to decorrelate with length and therefore focus more on the actual
  content. We then discard the length head in RL to prevent reward hacking on
  length. Experiments demonstrate that our approach almost eliminates the reward
  correlation with length, and improves the obtained policy by a significant
  margin.
paper_ai_summary: An evaluation protocol and reward model improvement address length bias in Reinforcement Learning from Human Feedback for LLMs.
paper_ai_keywords: Reinforcement Learning from Human Feedback (RLHF) reward hacking response length reward models evaluation protocol training configuraton hyperparameters feature representations linear heads policy