title: Pitfalls of Rule- and Model-based Verifiers -- A Case Study on
    Mathematical Reasoning
summary: Trustworthy verifiers are essential for the success of reinforcement learning
  with verifiable reward (RLVR), which is the core methodology behind various
  large reasoning models such as DeepSeek-R1. In complex domains like
  mathematical reasoning, rule-based verifiers have been widely adopted in
  previous works to train strong reasoning models. However, the reliability of
  these verifiers and their impact on the RL training process remain poorly
  understood. In this work, we take mathematical reasoning as a case study and
  conduct a comprehensive analysis of various verifiers in both static evaluation
  and RL training scenarios. First, we find that current open-source rule-based
  verifiers often fail to recognize equivalent answers presented in different
  formats across multiple commonly used mathematical datasets, resulting in
  non-negligible false negative rates. This limitation adversely affects RL
  training performance and becomes more pronounced as the policy model gets
  stronger. Subsequently, we investigate model-based verifiers as a potential
  solution to address these limitations. While the static evaluation shows that
  model-based verifiers achieve significantly higher verification accuracy,
  further analysis and RL training results imply that they are highly susceptible
  to hacking, where they misclassify certain patterns in responses as correct
  (i.e., false positives). This vulnerability is exploited during policy model
  optimization, leading to artificially inflated rewards. Our findings underscore
  the unique risks inherent to both rule-based and model-based verifiers, aiming
  to offer valuable insights to develop more robust reward systems in
  reinforcement learning.
paper_authors._id: 6837dcc41448b8bf0c91fa30
paper_authors.name: Yuzhen Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T07:41:53.737Z
paper_authors._id: 6462def82a83863b97c0611e
paper_authors.avatarUrl: /avatars/c03e9cc7d75b0266fcc56ecb6ee62148.svg
paper_authors.fullname: Yuzhen Huang
paper_authors.type: user
paper_authors.user: yuzhen17
paper_authors._id: 6837dcc41448b8bf0c91fa31
paper_authors.name: Weihao Zeng
paper_authors._id: 6837dcc41448b8bf0c91fa32
paper_authors.name: Xingshan Zeng
paper_authors._id: 6837dcc41448b8bf0c91fa33
paper_authors.name: Qi Zhu
paper_authors._id: 6837dcc41448b8bf0c91fa34
paper_authors.name: Junxian He
paper_title: Pitfalls of Rule- and Model-based Verifiers -- A Case Study on
    Mathematical Reasoning
paper_summary: Trustworthy verifiers are essential for the success of reinforcement learning
  with verifiable reward (RLVR), which is the core methodology behind various
  large reasoning models such as DeepSeek-R1. In complex domains like
  mathematical reasoning, rule-based verifiers have been widely adopted in
  previous works to train strong reasoning models. However, the reliability of
  these verifiers and their impact on the RL training process remain poorly
  understood. In this work, we take mathematical reasoning as a case study and
  conduct a comprehensive analysis of various verifiers in both static evaluation
  and RL training scenarios. First, we find that current open-source rule-based
  verifiers often fail to recognize equivalent answers presented in different
  formats across multiple commonly used mathematical datasets, resulting in
  non-negligible false negative rates. This limitation adversely affects RL
  training performance and becomes more pronounced as the policy model gets
  stronger. Subsequently, we investigate model-based verifiers as a potential
  solution to address these limitations. While the static evaluation shows that
  model-based verifiers achieve significantly higher verification accuracy,
  further analysis and RL training results imply that they are highly susceptible
  to hacking, where they misclassify certain patterns in responses as correct
  (i.e., false positives). This vulnerability is exploited during policy model
  optimization, leading to artificially inflated rewards. Our findings underscore
  the unique risks inherent to both rule-based and model-based verifiers, aiming
  to offer valuable insights to develop more robust reward systems in
  reinforcement learning.
paper_ai_summary: The study examines the effectiveness and reliability of rule-based and model-based verifiers in reinforcement learning with verifiable reward, highlighting limitations and vulnerabilities in their use for mathematical reasoning tasks.
paper_ai_keywords: reinforcement learning verifiable reward DeepSeek-R1 mathematical reasoning rule-based verifiers reward systems model-based verifiers false negatives false positives