title: RM-Bench: Benchmarking Reward Models of Language Models with Subtlety
    and Style
summary: Reward models are critical in techniques like Reinforcement Learning from
  Human Feedback (RLHF) and Inference Scaling Laws, where they guide language
  model alignment and select optimal responses. Despite their importance,
  existing reward model benchmarks often evaluate models by asking them to
  distinguish between responses generated by models of varying power. However,
  this approach fails to assess reward models on subtle but critical content
  changes and variations in style, resulting in a low correlation with policy
  model performance. To this end, we introduce RM-Bench, a novel benchmark
  designed to evaluate reward models based on their sensitivity to subtle content
  differences and resistance to style biases. Extensive experiments demonstrate
  that RM-Bench strongly correlates with policy model performance, making it a
  reliable reference for selecting reward models to align language models
  effectively. We evaluate nearly 40 reward models on RM-Bench. Our results
  reveal that even state-of-the-art models achieve an average performance of only
  46.6%, which falls short of random-level accuracy (50%) when faced with style
  bias interference. These findings highlight the significant room for
  improvement in current reward models. Related code and data are available at
  https://github.com/THU-KEG/RM-Bench.
paper_authors._id: 671725cc2df05eaa5c2f5020
paper_authors.name: Yantao Liu
paper_authors._id: 671725cc2df05eaa5c2f5021
paper_authors.name: Zijun Yao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T07:56:57.790Z
paper_authors._id: 648c4b46e549be47af1aafcd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648c4b46e549be47af1aafcd/YgOHbmUM2EDM-lb7GdqXz.jpeg
paper_authors.fullname: Zijun
paper_authors.type: user
paper_authors.user: TranSirius
paper_authors._id: 671725cc2df05eaa5c2f5022
paper_authors.name: Rui Min
paper_authors._id: 671725cc2df05eaa5c2f5023
paper_authors.name: Yixin Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T09:08:11.051Z
paper_authors._id: 6586ec334f349f95cf8c98d2
paper_authors.avatarUrl: /avatars/ff74ebb7d661ab70401d709885fb9717.svg
paper_authors.fullname: caoyixin
paper_authors.type: user
paper_authors.user: Nuomei
paper_authors._id: 671725cc2df05eaa5c2f5024
paper_authors.name: Lei Hou
paper_authors._id: 671725cc2df05eaa5c2f5025
paper_authors.name: Juanzi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T09:07:50.457Z
paper_authors._id: 65df8cbc2705d9672f55d1aa
paper_authors.avatarUrl: /avatars/63e46f15bb76bd9d4508fd0f54f39829.svg
paper_authors.fullname: Juanzi Li
paper_authors.type: user
paper_authors.user: juanli
paper_title: RM-Bench: Benchmarking Reward Models of Language Models with Subtlety
    and Style
paper_summary: Reward models are critical in techniques like Reinforcement Learning from
  Human Feedback (RLHF) and Inference Scaling Laws, where they guide language
  model alignment and select optimal responses. Despite their importance,
  existing reward model benchmarks often evaluate models by asking them to
  distinguish between responses generated by models of varying power. However,
  this approach fails to assess reward models on subtle but critical content
  changes and variations in style, resulting in a low correlation with policy
  model performance. To this end, we introduce RM-Bench, a novel benchmark
  designed to evaluate reward models based on their sensitivity to subtle content
  differences and resistance to style biases. Extensive experiments demonstrate
  that RM-Bench strongly correlates with policy model performance, making it a
  reliable reference for selecting reward models to align language models
  effectively. We evaluate nearly 40 reward models on RM-Bench. Our results
  reveal that even state-of-the-art models achieve an average performance of only
  46.6%, which falls short of random-level accuracy (50%) when faced with style
  bias interference. These findings highlight the significant room for
  improvement in current reward models. Related code and data are available at
  https://github.com/THU-KEG/RM-Bench.
paper_ai_summary: RM-Bench evaluates reward models based on sensitivity to subtle content differences and style biases, providing a reliable metric for selecting reward models in RLHF and Inference Scaling Laws.
paper_ai_keywords: reward models Reinforcement Learning from Human Feedback (RLHF) Inference Scaling Laws language model alignment RM-Bench reward model performance policy model performance style bias