title: TinyV: Reducing False Negatives in Verification Improves RL for LLM
    Reasoning
summary: Reinforcement Learning (RL) has become a powerful tool for enhancing the
  reasoning abilities of large language models (LLMs) by optimizing their
  policies with reward signals. Yet, RL's success relies on the reliability of
  rewards, which are provided by verifiers. In this paper, we expose and analyze
  a widespread problem--false negatives--where verifiers wrongly reject correct
  model outputs. Our in-depth study of the Big-Math-RL-Verified dataset reveals
  that over 38% of model-generated responses suffer from false negatives, where
  the verifier fails to recognize correct answers. We show, both empirically and
  theoretically, that these false negatives severely impair RL training by
  depriving the model of informative gradient signals and slowing convergence. To
  mitigate this, we propose tinyV, a lightweight LLM-based verifier that augments
  existing rule-based methods, which dynamically identifies potential false
  negatives and recovers valid responses to produce more accurate reward
  estimates. Across multiple math-reasoning benchmarks, integrating TinyV boosts
  pass rates by up to 10% and accelerates convergence relative to the baseline.
  Our findings highlight the critical importance of addressing verifier false
  negatives and offer a practical approach to improve RL-based fine-tuning of
  LLMs. Our code is available at https://github.com/uw-nsl/TinyV.
paper_authors._id: 682fdb318df2d5446a1cf30b
paper_authors.name: Zhangchen Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T07:44:39.703Z
paper_authors._id: 653df1323479e9ebbe3eb6cc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/653df1323479e9ebbe3eb6cc/K_g-r1iMRNKj99LXPuYF3.jpeg
paper_authors.fullname: Zhangchen Xu
paper_authors.type: user
paper_authors.user: zhangchenxu
paper_authors._id: 682fdb318df2d5446a1cf30c
paper_authors.name: Yuetai Li
paper_authors._id: 682fdb318df2d5446a1cf30d
paper_authors.name: Fengqing Jiang
paper_authors._id: 682fdb318df2d5446a1cf30e
paper_authors.name: Bhaskar Ramasubramanian
paper_authors._id: 682fdb318df2d5446a1cf30f
paper_authors.name: Luyao Niu
paper_authors._id: 682fdb318df2d5446a1cf310
paper_authors.name: Bill Yuchen Lin
paper_authors._id: 682fdb318df2d5446a1cf311
paper_authors.name: Radha Poovendran
paper_title: TinyV: Reducing False Negatives in Verification Improves RL for LLM
    Reasoning
paper_summary: Reinforcement Learning (RL) has become a powerful tool for enhancing the
  reasoning abilities of large language models (LLMs) by optimizing their
  policies with reward signals. Yet, RL's success relies on the reliability of
  rewards, which are provided by verifiers. In this paper, we expose and analyze
  a widespread problem--false negatives--where verifiers wrongly reject correct
  model outputs. Our in-depth study of the Big-Math-RL-Verified dataset reveals
  that over 38% of model-generated responses suffer from false negatives, where
  the verifier fails to recognize correct answers. We show, both empirically and
  theoretically, that these false negatives severely impair RL training by
  depriving the model of informative gradient signals and slowing convergence. To
  mitigate this, we propose tinyV, a lightweight LLM-based verifier that augments
  existing rule-based methods, which dynamically identifies potential false
  negatives and recovers valid responses to produce more accurate reward
  estimates. Across multiple math-reasoning benchmarks, integrating TinyV boosts
  pass rates by up to 10% and accelerates convergence relative to the baseline.
  Our findings highlight the critical importance of addressing verifier false
  negatives and offer a practical approach to improve RL-based fine-tuning of
  LLMs. Our code is available at https://github.com/uw-nsl/TinyV.
paper_ai_summary: TinyV, a lightweight LLM-based verifier, improves RL training of large language models by addressing false negatives from existing rule-based verifiers, enhancing reward accuracy and convergence speed.
paper_ai_keywords: Reinforcement Learning large language models policies reward signals verifiers false negatives Big-Math-RL-Verified dataset gradient signals convergence TinyV rule-based methods math-reasoning benchmarks pass rates