Taming Overconfidence in LLMs: Reward Calibration in RLHF
Language model calibration refers to the alignment between the confidence of
the model and the actual performance of its responses. While previous studies
point out the overconfidence phenomenon in Large Language Models (LLMs) and
show that LLMs trained with Reinforcement Learning from Human Feedback (RLHF)
are overconfident with a more sharpened output probability, in this study, we
reveal that RLHF tends to lead models to express verbalized overconfidence in
their own responses. We investigate the underlying cause of this overconfidence
and demonstrate that reward models used for Proximal Policy Optimization (PPO)
exhibit inherent biases towards high-confidence scores regardless of the actual
quality of responses. Building upon this insight, we propose two PPO variants:
PPO-M: PPO with Calibrated Reward Modeling and PPO-C: PPO with Calibrated
Reward Calculation. PPO-M integrates explicit confidence scores in reward model
training, which calibrates reward models to better capture the alignment
between response quality and verbalized confidence. PPO-C adjusts the reward
score during PPO based on the difference between the current reward and the
moving average of past rewards. Both PPO-M and PPO-C can be seamlessly
integrated into the current PPO pipeline and do not require additional golden
labels. We evaluate our methods on both Llama3-8B and Mistral-7B across six
diverse datasets including multiple-choice and open-ended generation.
Experiment results demonstrate that both of our methods can reduce calibration
error and maintain performance comparable to standard PPO. We further show that
they do not compromise model capabilities in open-ended conversation settings.
67107fcbf1279fe0dfcfbe96
Jixuan Leng
claimed_verified
2024-10-18T08:14:25.715Z
64811214cacb1c4a06988bd9
/avatars/538a3029878c2200df049b54c97c9327.svg
Jixuan Leng
user
JixuanLeng
67107fcbf1279fe0dfcfbe97
Chengsong Huang
extracted_pending
2024-10-17T03:09:09.055Z
647b8885aba7062fe5c32000
/avatars/184e6199e20ad63b3de12436bdd00a00.svg
Banghua Zhu
user
banghua
67107fcbf1279fe0dfcfbe98
Banghua Zhu
67107fcbf1279fe0dfcfbe99
Jiaxin Huang
claimed_verified
2024-10-17T11:24:34.435Z
64efbf39b3610349e84db417
/avatars/9e09a20e88f8cf5ce119efc0dadc3b7b.svg
Jiaxin Huang
user
teapot123
Taming Overconfidence in LLMs: Reward Calibration in RLHF
Language model calibration refers to the alignment between the confidence of
the model and the actual performance of its responses. While previous studies
point out the overconfidence phenomenon in Large Language Models (LLMs) and
show that LLMs trained with Reinforcement Learning from Human Feedback (RLHF)
are overconfident with a more sharpened output probability, in this study, we
reveal that RLHF tends to lead models to express verbalized overconfidence in
their own responses. We investigate the underlying cause of this overconfidence
and demonstrate that reward models used for Proximal Policy Optimization (PPO)
exhibit inherent biases towards high-confidence scores regardless of the actual
quality of responses. Building upon this insight, we propose two PPO variants:
PPO-M: PPO with Calibrated Reward Modeling and PPO-C: PPO with Calibrated
Reward Calculation. PPO-M integrates explicit confidence scores in reward model
training, which calibrates reward models to better capture the alignment
between response quality and verbalized confidence. PPO-C adjusts the reward
score during PPO based on the difference between the current reward and the
moving average of past rewards. Both PPO-M and PPO-C can be seamlessly
integrated into the current PPO pipeline and do not require additional golden
labels. We evaluate our methods on both Llama3-8B and Mistral-7B across six
diverse datasets including multiple-choice and open-ended generation.
Experiment results demonstrate that both of our methods can reduce calibration
error and maintain performance comparable to standard PPO. We further show that
they do not compromise model capabilities in open-ended conversation settings.
Proposes PPO-M and PPO-C to reduce overconfidence in LLMs trained with RLHF by calibrating reward models and reward scores.
Reinforcement Learning from Human Feedback (RLHF) Proximal Policy Optimization (PPO) Llama3-8B Mistral-7B multiple-choice open-ended generation open-ended conversation