title: RRM: Robust Reward Model Training Mitigates Reward Hacking
summary: Reward models (RMs) play a pivotal role in aligning large language models
  (LLMs) with human preferences. However, traditional RM training, which relies
  on response pairs tied to specific prompts, struggles to disentangle
  prompt-driven preferences from prompt-independent artifacts, such as response
  length and format. In this work, we expose a fundamental limitation of current
  RM training methods, where RMs fail to effectively distinguish between
  contextual signals and irrelevant artifacts when determining preferences. To
  address this, we introduce a causal framework that learns preferences
  independent of these artifacts and propose a novel data augmentation technique
  designed to eliminate them. Extensive experiments show that our approach
  successfully filters out undesirable artifacts, yielding a more robust reward
  model (RRM). Our RRM improves the performance of a pairwise reward model
  trained on Gemma-2-9b-it, on RewardBench, increasing accuracy from 80.61% to
  84.15%. Additionally, we train two DPO policies using both the RM and RRM,
  demonstrating that the RRM significantly enhances DPO-aligned policies,
  improving MT-Bench scores from 7.27 to 8.31 and length-controlled win-rates in
  AlpacaEval-2 from 33.46% to 52.49%.
paper_authors._id: 66f426f30c1433d7a076bfff
paper_authors.name: Tianqi Liu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-09-25T15:35:37.888Z
paper_authors._id: 646784af696e7355f5d5001f
paper_authors.avatarUrl: /avatars/39ce8999c296584ee0bdeb7848eee6d9.svg
paper_authors.fullname: Tianqi Liu
paper_authors.type: user
paper_authors.user: TianqiLiuAI
paper_authors._id: 66f426f30c1433d7a076c000
paper_authors.name: Wei Xiong
paper_authors._id: 66f426f30c1433d7a076c001
paper_authors.name: Jie Ren
paper_authors._id: 66f426f30c1433d7a076c002
paper_authors.name: Lichang Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-26T08:23:29.589Z
paper_authors._id: 62323bb408bcea92917e42ee
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62323bb408bcea92917e42ee/2vHxkv-oSROtLteOnqa8P.jpeg
paper_authors.fullname: Lichang Chen
paper_authors.type: user
paper_authors.user: Lichang-Chen
paper_authors._id: 66f426f30c1433d7a076c003
paper_authors.name: Junru Wu
paper_authors._id: 66f426f30c1433d7a076c004
paper_authors.name: Rishabh Joshi
paper_authors._id: 66f426f30c1433d7a076c005
paper_authors.name: Yang Gao
paper_authors._id: 66f426f30c1433d7a076c006
paper_authors.name: Jiaming Shen
paper_authors._id: 66f426f30c1433d7a076c007
paper_authors.name: Zhen Qin
paper_authors._id: 66f426f30c1433d7a076c008
paper_authors.name: Tianhe Yu
paper_authors._id: 66f426f30c1433d7a076c009
paper_authors.name: Daniel Sohn
paper_authors._id: 66f426f30c1433d7a076c00a
paper_authors.name: Anastasiia Makarova
paper_authors._id: 66f426f30c1433d7a076c00b
paper_authors.name: Jeremiah Liu
paper_authors._id: 66f426f30c1433d7a076c00c
paper_authors.name: Yuan Liu
paper_authors._id: 66f426f30c1433d7a076c00d
paper_authors.name: Bilal Piot
paper_authors._id: 66f426f30c1433d7a076c00e
paper_authors.name: Abe Ittycheriah
paper_authors._id: 66f426f30c1433d7a076c00f
paper_authors.name: Aviral Kumar
paper_authors._id: 66f426f30c1433d7a076c010
paper_authors.name: Mohammad Saleh
paper_title: RRM: Robust Reward Model Training Mitigates Reward Hacking
paper_summary: Reward models (RMs) play a pivotal role in aligning large language models
  (LLMs) with human preferences. However, traditional RM training, which relies
  on response pairs tied to specific prompts, struggles to disentangle
  prompt-driven preferences from prompt-independent artifacts, such as response
  length and format. In this work, we expose a fundamental limitation of current
  RM training methods, where RMs fail to effectively distinguish between
  contextual signals and irrelevant artifacts when determining preferences. To
  address this, we introduce a causal framework that learns preferences
  independent of these artifacts and propose a novel data augmentation technique
  designed to eliminate them. Extensive experiments show that our approach
  successfully filters out undesirable artifacts, yielding a more robust reward
  model (RRM). Our RRM improves the performance of a pairwise reward model
  trained on Gemma-2-9b-it, on RewardBench, increasing accuracy from 80.61% to
  84.15%. Additionally, we train two DPO policies using both the RM and RRM,
  demonstrating that the RRM significantly enhances DPO-aligned policies,
  improving MT-Bench scores from 7.27 to 8.31 and length-controlled win-rates in
  AlpacaEval-2 from 33.46% to 52.49%.
paper_ai_keywords: Reward models (RMs) Large language models (LLMs) Response pairs Prompt-driven preferences Prompt-independent artifacts Response length Response format Causal framework Data augmentation Phrase embeddings Robust reward model (RRM) Pairwise reward model Gemma-2-9b RewardBench DPO policies MT-Bench scores AlpacaEval-2 Length-controlled win-rates