RRM: Robust Reward Model Training Mitigates Reward Hacking
Reward models (RMs) play a pivotal role in aligning large language models
(LLMs) with human preferences. However, traditional RM training, which relies
on response pairs tied to specific prompts, struggles to disentangle
prompt-driven preferences from prompt-independent artifacts, such as response
length and format. In this work, we expose a fundamental limitation of current
RM training methods, where RMs fail to effectively distinguish between
contextual signals and irrelevant artifacts when determining preferences. To
address this, we introduce a causal framework that learns preferences
independent of these artifacts and propose a novel data augmentation technique
designed to eliminate them. Extensive experiments show that our approach
successfully filters out undesirable artifacts, yielding a more robust reward
model (RRM). Our RRM improves the performance of a pairwise reward model
trained on Gemma-2-9b-it, on RewardBench, increasing accuracy from 80.61% to
84.15%. Additionally, we train two DPO policies using both the RM and RRM,
demonstrating that the RRM significantly enhances DPO-aligned policies,
improving MT-Bench scores from 7.27 to 8.31 and length-controlled win-rates in
AlpacaEval-2 from 33.46% to 52.49%.
66f426f30c1433d7a076bfff
Tianqi Liu
extracted_confirmed
2024-09-25T15:35:37.888Z
646784af696e7355f5d5001f
/avatars/39ce8999c296584ee0bdeb7848eee6d9.svg
Tianqi Liu
user
TianqiLiuAI
66f426f30c1433d7a076c000
Wei Xiong
66f426f30c1433d7a076c001
Jie Ren
66f426f30c1433d7a076c002
Lichang Chen
claimed_verified
2024-09-26T08:23:29.589Z
62323bb408bcea92917e42ee
https://cdn-avatars.huggingface.co/v1/production/uploads/62323bb408bcea92917e42ee/2vHxkv-oSROtLteOnqa8P.jpeg
Lichang Chen
user
Lichang-Chen
66f426f30c1433d7a076c003
Junru Wu
66f426f30c1433d7a076c004
Rishabh Joshi
66f426f30c1433d7a076c005
Yang Gao
66f426f30c1433d7a076c006
Jiaming Shen
66f426f30c1433d7a076c007
Zhen Qin
66f426f30c1433d7a076c008
Tianhe Yu
66f426f30c1433d7a076c009
Daniel Sohn
66f426f30c1433d7a076c00a
Anastasiia Makarova
66f426f30c1433d7a076c00b
Jeremiah Liu
66f426f30c1433d7a076c00c
Yuan Liu
66f426f30c1433d7a076c00d
Bilal Piot
66f426f30c1433d7a076c00e
Abe Ittycheriah
66f426f30c1433d7a076c00f
Aviral Kumar
66f426f30c1433d7a076c010
Mohammad Saleh
RRM: Robust Reward Model Training Mitigates Reward Hacking
Reward models (RMs) play a pivotal role in aligning large language models
(LLMs) with human preferences. However, traditional RM training, which relies
on response pairs tied to specific prompts, struggles to disentangle
prompt-driven preferences from prompt-independent artifacts, such as response
length and format. In this work, we expose a fundamental limitation of current
RM training methods, where RMs fail to effectively distinguish between
contextual signals and irrelevant artifacts when determining preferences. To
address this, we introduce a causal framework that learns preferences
independent of these artifacts and propose a novel data augmentation technique
designed to eliminate them. Extensive experiments show that our approach
successfully filters out undesirable artifacts, yielding a more robust reward
model (RRM). Our RRM improves the performance of a pairwise reward model
trained on Gemma-2-9b-it, on RewardBench, increasing accuracy from 80.61% to
84.15%. Additionally, we train two DPO policies using both the RM and RRM,
demonstrating that the RRM significantly enhances DPO-aligned policies,
improving MT-Bench scores from 7.27 to 8.31 and length-controlled win-rates in
AlpacaEval-2 from 33.46% to 52.49%.
Introducing a causal framework and data augmentation for reward models significantly improves their robustness and alignment with human preferences.
reward models large language models RM training response pairs prompt-driven preferences prompt-independent artifacts contextual signals irrelevant artifacts robust reward model pairwise reward model Gemma-2-9b-it RewardBench DPO policies MT-Bench AlpacaEval-2