title: Taming Overconfidence in LLMs: Reward Calibration in RLHF
summary: Language model calibration refers to the alignment between the confidence of
  the model and the actual performance of its responses. While previous studies
  point out the overconfidence phenomenon in Large Language Models (LLMs) and
  show that LLMs trained with Reinforcement Learning from Human Feedback (RLHF)
  are overconfident with a more sharpened output probability, in this study, we
  reveal that RLHF tends to lead models to express verbalized overconfidence in
  their own responses. We investigate the underlying cause of this overconfidence
  and demonstrate that reward models used for Proximal Policy Optimization (PPO)
  exhibit inherent biases towards high-confidence scores regardless of the actual
  quality of responses. Building upon this insight, we propose two PPO variants:
  PPO-M: PPO with Calibrated Reward Modeling and PPO-C: PPO with Calibrated
  Reward Calculation. PPO-M integrates explicit confidence scores in reward model
  training, which calibrates reward models to better capture the alignment
  between response quality and verbalized confidence. PPO-C adjusts the reward
  score during PPO based on the difference between the current reward and the
  moving average of past rewards. Both PPO-M and PPO-C can be seamlessly
  integrated into the current PPO pipeline and do not require additional golden
  labels. We evaluate our methods on both Llama3-8B and Mistral-7B across six
  diverse datasets including multiple-choice and open-ended generation.
  Experiment results demonstrate that both of our methods can reduce calibration
  error and maintain performance comparable to standard PPO. We further show that
  they do not compromise model capabilities in open-ended conversation settings.
paper_authors._id: 67107fcbf1279fe0dfcfbe96
paper_authors.name: Jixuan Leng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T08:14:25.715Z
paper_authors._id: 64811214cacb1c4a06988bd9
paper_authors.avatarUrl: /avatars/538a3029878c2200df049b54c97c9327.svg
paper_authors.fullname: Jixuan Leng
paper_authors.type: user
paper_authors.user: JixuanLeng
paper_authors._id: 67107fcbf1279fe0dfcfbe97
paper_authors.name: Chengsong Huang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-10-17T03:09:09.055Z
paper_authors._id: 647b8885aba7062fe5c32000
paper_authors.avatarUrl: /avatars/128ceae78490110ae41202851e84d58e.svg
paper_authors.fullname: Banghua Zhu
paper_authors.type: user
paper_authors.user: banghua
paper_authors._id: 67107fcbf1279fe0dfcfbe98
paper_authors.name: Banghua Zhu
paper_authors._id: 67107fcbf1279fe0dfcfbe99
paper_authors.name: Jiaxin Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-17T11:24:34.435Z
paper_authors._id: 64efbf39b3610349e84db417
paper_authors.avatarUrl: /avatars/9e09a20e88f8cf5ce119efc0dadc3b7b.svg
paper_authors.fullname: Jiaxin Huang
paper_authors.type: user
paper_authors.user: teapot123
paper_title: Taming Overconfidence in LLMs: Reward Calibration in RLHF
paper_summary: Language model calibration refers to the alignment between the confidence of
  the model and the actual performance of its responses. While previous studies
  point out the overconfidence phenomenon in Large Language Models (LLMs) and
  show that LLMs trained with Reinforcement Learning from Human Feedback (RLHF)
  are overconfident with a more sharpened output probability, in this study, we
  reveal that RLHF tends to lead models to express verbalized overconfidence in
  their own responses. We investigate the underlying cause of this overconfidence
  and demonstrate that reward models used for Proximal Policy Optimization (PPO)
  exhibit inherent biases towards high-confidence scores regardless of the actual
  quality of responses. Building upon this insight, we propose two PPO variants:
  PPO-M: PPO with Calibrated Reward Modeling and PPO-C: PPO with Calibrated
  Reward Calculation. PPO-M integrates explicit confidence scores in reward model
  training, which calibrates reward models to better capture the alignment
  between response quality and verbalized confidence. PPO-C adjusts the reward
  score during PPO based on the difference between the current reward and the
  moving average of past rewards. Both PPO-M and PPO-C can be seamlessly
  integrated into the current PPO pipeline and do not require additional golden
  labels. We evaluate our methods on both Llama3-8B and Mistral-7B across six
  diverse datasets including multiple-choice and open-ended generation.
  Experiment results demonstrate that both of our methods can reduce calibration
  error and maintain performance comparable to standard PPO. We further show that
  they do not compromise model capabilities in open-ended conversation settings.
paper_ai_summary: Proposes PPO-M and PPO-C to reduce overconfidence in LLMs trained with RLHF by calibrating reward models and reward scores.
paper_ai_keywords: Reinforcement Learning from Human Feedback (RLHF) Proximal Policy Optimization (PPO) Llama3-8B Mistral-7B multiple-choice open-ended generation open-ended conversation