VerifyBench: Benchmarking Reference-based Reward Systems for Large
  Language Models
Large reasoning models such as OpenAI o1 and DeepSeek-R1 have achieved
remarkable performance in the domain of reasoning. A key component of their
training is the incorporation of verifiable rewards within reinforcement
learning (RL). However, existing reward benchmarks do not evaluate
reference-based reward systems, leaving researchers with limited understanding
of the accuracy of verifiers used in RL. In this paper, we introduce two
benchmarks, VerifyBench and VerifyBench-Hard, designed to assess the
performance of reference-based reward systems. These benchmarks are constructed
through meticulous data collection and curation, followed by careful human
annotation to ensure high quality. Current models still show considerable room
for improvement on both VerifyBench and VerifyBench-Hard, especially
smaller-scale models. Furthermore, we conduct a thorough and comprehensive
analysis of evaluation results, offering insights for understanding and
developing reference-based reward systems. Our proposed benchmarks serve as
effective tools for guiding the development of verifier accuracy and the
reasoning capabilities of models trained via RL in reasoning tasks.
682f4bed557039a63173829f
Yuchen Yan
claimed_verified
2025-05-26T08:17:16.142Z
64098738342c26884c792c93
https://cdn-avatars.huggingface.co/v1/production/uploads/64098738342c26884c792c93/SxBUd-wLrl-PjQsrVYJte.jpeg
Yuchen Yan
user
yanyc
682f4bed557039a6317382a0
Jin Jiang
682f4bed557039a6317382a1
Zhenbang Ren
682f4bed557039a6317382a2
Yijun Li
682f4bed557039a6317382a3
Xudong Cai
682f4bed557039a6317382a4
Yang Liu
682f4bed557039a6317382a5
Xin Xu
682f4bed557039a6317382a6
Mengdi Zhang
682f4bed557039a6317382a7
Jian Shao
682f4bed557039a6317382a8
Yongliang Shen
claimed_verified
2025-05-26T08:17:13.646Z
5e1058e9fcf41d740b69966d
/avatars/ce74839ba871f2b54313a670a233ba82.svg
Yongliang Shen
user
tricktreat
682f4bed557039a6317382a9
Jun Xiao
682f4bed557039a6317382aa
Yueting Zhuang
VerifyBench: Benchmarking Reference-based Reward Systems for Large
  Language Models
Large reasoning models such as OpenAI o1 and DeepSeek-R1 have achieved
remarkable performance in the domain of reasoning. A key component of their
training is the incorporation of verifiable rewards within reinforcement
learning (RL). However, existing reward benchmarks do not evaluate
reference-based reward systems, leaving researchers with limited understanding
of the accuracy of verifiers used in RL. In this paper, we introduce two
benchmarks, VerifyBench and VerifyBench-Hard, designed to assess the
performance of reference-based reward systems. These benchmarks are constructed
through meticulous data collection and curation, followed by careful human
annotation to ensure high quality. Current models still show considerable room
for improvement on both VerifyBench and VerifyBench-Hard, especially
smaller-scale models. Furthermore, we conduct a thorough and comprehensive
analysis of evaluation results, offering insights for understanding and
developing reference-based reward systems. Our proposed benchmarks serve as
effective tools for guiding the development of verifier accuracy and the
reasoning capabilities of models trained via RL in reasoning tasks.
Two new benchmarks, VerifyBench and VerifyBench-Hard, are introduced to evaluate the accuracy of reference-based reward systems in reinforcement learning for reasoning tasks.
reinforcement learning RL reward benchmarks reference-based reward systems verifiable rewards reasoning models OpenAI o1 DeepSeek-R1 human annotation model evaluation verifier accuracy