title: VerifyBench: Benchmarking Reference-based Reward Systems for Large
    Language Models
summary: Large reasoning models such as OpenAI o1 and DeepSeek-R1 have achieved
  remarkable performance in the domain of reasoning. A key component of their
  training is the incorporation of verifiable rewards within reinforcement
  learning (RL). However, existing reward benchmarks do not evaluate
  reference-based reward systems, leaving researchers with limited understanding
  of the accuracy of verifiers used in RL. In this paper, we introduce two
  benchmarks, VerifyBench and VerifyBench-Hard, designed to assess the
  performance of reference-based reward systems. These benchmarks are constructed
  through meticulous data collection and curation, followed by careful human
  annotation to ensure high quality. Current models still show considerable room
  for improvement on both VerifyBench and VerifyBench-Hard, especially
  smaller-scale models. Furthermore, we conduct a thorough and comprehensive
  analysis of evaluation results, offering insights for understanding and
  developing reference-based reward systems. Our proposed benchmarks serve as
  effective tools for guiding the development of verifier accuracy and the
  reasoning capabilities of models trained via RL in reasoning tasks.
paper_authors._id: 682f4bed557039a63173829f
paper_authors.name: Yuchen Yan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:17:16.142Z
paper_authors._id: 64098738342c26884c792c93
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64098738342c26884c792c93/SxBUd-wLrl-PjQsrVYJte.jpeg
paper_authors.fullname: Yuchen Yan
paper_authors.type: user
paper_authors.user: yanyc
paper_authors._id: 682f4bed557039a6317382a0
paper_authors.name: Jin Jiang
paper_authors._id: 682f4bed557039a6317382a1
paper_authors.name: Zhenbang Ren
paper_authors._id: 682f4bed557039a6317382a2
paper_authors.name: Yijun Li
paper_authors._id: 682f4bed557039a6317382a3
paper_authors.name: Xudong Cai
paper_authors._id: 682f4bed557039a6317382a4
paper_authors.name: Yang Liu
paper_authors._id: 682f4bed557039a6317382a5
paper_authors.name: Xin Xu
paper_authors._id: 682f4bed557039a6317382a6
paper_authors.name: Mengdi Zhang
paper_authors._id: 682f4bed557039a6317382a7
paper_authors.name: Jian Shao
paper_authors._id: 682f4bed557039a6317382a8
paper_authors.name: Yongliang Shen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:17:13.646Z
paper_authors._id: 5e1058e9fcf41d740b69966d
paper_authors.avatarUrl: /avatars/ce74839ba871f2b54313a670a233ba82.svg
paper_authors.fullname: Yongliang Shen
paper_authors.type: user
paper_authors.user: tricktreat
paper_authors._id: 682f4bed557039a6317382a9
paper_authors.name: Jun Xiao
paper_authors._id: 682f4bed557039a6317382aa
paper_authors.name: Yueting Zhuang
paper_title: VerifyBench: Benchmarking Reference-based Reward Systems for Large
    Language Models
paper_summary: Large reasoning models such as OpenAI o1 and DeepSeek-R1 have achieved
  remarkable performance in the domain of reasoning. A key component of their
  training is the incorporation of verifiable rewards within reinforcement
  learning (RL). However, existing reward benchmarks do not evaluate
  reference-based reward systems, leaving researchers with limited understanding
  of the accuracy of verifiers used in RL. In this paper, we introduce two
  benchmarks, VerifyBench and VerifyBench-Hard, designed to assess the
  performance of reference-based reward systems. These benchmarks are constructed
  through meticulous data collection and curation, followed by careful human
  annotation to ensure high quality. Current models still show considerable room
  for improvement on both VerifyBench and VerifyBench-Hard, especially
  smaller-scale models. Furthermore, we conduct a thorough and comprehensive
  analysis of evaluation results, offering insights for understanding and
  developing reference-based reward systems. Our proposed benchmarks serve as
  effective tools for guiding the development of verifier accuracy and the
  reasoning capabilities of models trained via RL in reasoning tasks.
paper_ai_summary: Two new benchmarks, VerifyBench and VerifyBench-Hard, are introduced to evaluate the accuracy of reference-based reward systems in reinforcement learning for reasoning tasks.
paper_ai_keywords: reinforcement learning RL reward benchmarks reference-based reward systems verifiable rewards reasoning models OpenAI o1 DeepSeek-R1 human annotation model evaluation verifier accuracy