RM-Bench: Benchmarking Reward Models of Language Models with Subtlety
  and Style
Reward models are critical in techniques like Reinforcement Learning from
Human Feedback (RLHF) and Inference Scaling Laws, where they guide language
model alignment and select optimal responses. Despite their importance,
existing reward model benchmarks often evaluate models by asking them to
distinguish between responses generated by models of varying power. However,
this approach fails to assess reward models on subtle but critical content
changes and variations in style, resulting in a low correlation with policy
model performance. To this end, we introduce RM-Bench, a novel benchmark
designed to evaluate reward models based on their sensitivity to subtle content
differences and resistance to style biases. Extensive experiments demonstrate
that RM-Bench strongly correlates with policy model performance, making it a
reliable reference for selecting reward models to align language models
effectively. We evaluate nearly 40 reward models on RM-Bench. Our results
reveal that even state-of-the-art models achieve an average performance of only
46.6%, which falls short of random-level accuracy (50%) when faced with style
bias interference. These findings highlight the significant room for
improvement in current reward models. Related code and data are available at
https://github.com/THU-KEG/RM-Bench.
671725cc2df05eaa5c2f5020
Yantao Liu
671725cc2df05eaa5c2f5021
Zijun Yao
claimed_verified
2024-10-22T07:56:57.790Z
648c4b46e549be47af1aafcd
https://cdn-avatars.huggingface.co/v1/production/uploads/648c4b46e549be47af1aafcd/YgOHbmUM2EDM-lb7GdqXz.jpeg
Zijun
user
TranSirius
671725cc2df05eaa5c2f5022
Rui Min
671725cc2df05eaa5c2f5023
Yixin Cao
admin_assigned
2024-10-22T09:08:11.051Z
6586ec334f349f95cf8c98d2
/avatars/ff74ebb7d661ab70401d709885fb9717.svg
caoyixin
user
Nuomei
671725cc2df05eaa5c2f5024
Lei Hou
671725cc2df05eaa5c2f5025
Juanzi Li
admin_assigned
2024-10-22T09:07:50.457Z
65df8cbc2705d9672f55d1aa
/avatars/63e46f15bb76bd9d4508fd0f54f39829.svg
Juanzi Li
user
juanli
RM-Bench: Benchmarking Reward Models of Language Models with Subtlety
  and Style
Reward models are critical in techniques like Reinforcement Learning from
Human Feedback (RLHF) and Inference Scaling Laws, where they guide language
model alignment and select optimal responses. Despite their importance,
existing reward model benchmarks often evaluate models by asking them to
distinguish between responses generated by models of varying power. However,
this approach fails to assess reward models on subtle but critical content
changes and variations in style, resulting in a low correlation with policy
model performance. To this end, we introduce RM-Bench, a novel benchmark
designed to evaluate reward models based on their sensitivity to subtle content
differences and resistance to style biases. Extensive experiments demonstrate
that RM-Bench strongly correlates with policy model performance, making it a
reliable reference for selecting reward models to align language models
effectively. We evaluate nearly 40 reward models on RM-Bench. Our results
reveal that even state-of-the-art models achieve an average performance of only
46.6%, which falls short of random-level accuracy (50%) when faced with style
bias interference. These findings highlight the significant room for
improvement in current reward models. Related code and data are available at
https://github.com/THU-KEG/RM-Bench.
RM-Bench evaluates reward models based on sensitivity to subtle content differences and style biases, providing a reliable metric for selecting reward models in RLHF and Inference Scaling Laws.
reward models Reinforcement Learning from Human Feedback (RLHF) Inference Scaling Laws language model alignment RM-Bench reward model performance policy model performance style bias