TinyV: Reducing False Negatives in Verification Improves RL for LLM
  Reasoning
Reinforcement Learning (RL) has become a powerful tool for enhancing the
reasoning abilities of large language models (LLMs) by optimizing their
policies with reward signals. Yet, RL's success relies on the reliability of
rewards, which are provided by verifiers. In this paper, we expose and analyze
a widespread problem--false negatives--where verifiers wrongly reject correct
model outputs. Our in-depth study of the Big-Math-RL-Verified dataset reveals
that over 38% of model-generated responses suffer from false negatives, where
the verifier fails to recognize correct answers. We show, both empirically and
theoretically, that these false negatives severely impair RL training by
depriving the model of informative gradient signals and slowing convergence. To
mitigate this, we propose tinyV, a lightweight LLM-based verifier that augments
existing rule-based methods, which dynamically identifies potential false
negatives and recovers valid responses to produce more accurate reward
estimates. Across multiple math-reasoning benchmarks, integrating TinyV boosts
pass rates by up to 10% and accelerates convergence relative to the baseline.
Our findings highlight the critical importance of addressing verifier false
negatives and offer a practical approach to improve RL-based fine-tuning of
LLMs. Our code is available at https://github.com/uw-nsl/TinyV.
682fdb318df2d5446a1cf30b
Zhangchen Xu
claimed_verified
2025-05-29T07:44:39.703Z
653df1323479e9ebbe3eb6cc
https://cdn-avatars.huggingface.co/v1/production/uploads/653df1323479e9ebbe3eb6cc/K_g-r1iMRNKj99LXPuYF3.jpeg
Zhangchen Xu
user
zhangchenxu
682fdb318df2d5446a1cf30c
Yuetai Li
682fdb318df2d5446a1cf30d
Fengqing Jiang
682fdb318df2d5446a1cf30e
Bhaskar Ramasubramanian
682fdb318df2d5446a1cf30f
Luyao Niu
claimed_verified
2025-07-15T19:12:17.813Z
666dfd4770f5a2cb4aefd12f
/avatars/fa0e0dbc203a21e58dda8fdb4cbc67ad.svg
Luyao Niu
user
LNIU
682fdb318df2d5446a1cf310
Bill Yuchen Lin
682fdb318df2d5446a1cf311
Radha Poovendran
TinyV: Reducing False Negatives in Verification Improves RL for LLM
  Reasoning
Reinforcement Learning (RL) has become a powerful tool for enhancing the
reasoning abilities of large language models (LLMs) by optimizing their
policies with reward signals. Yet, RL's success relies on the reliability of
rewards, which are provided by verifiers. In this paper, we expose and analyze
a widespread problem--false negatives--where verifiers wrongly reject correct
model outputs. Our in-depth study of the Big-Math-RL-Verified dataset reveals
that over 38% of model-generated responses suffer from false negatives, where
the verifier fails to recognize correct answers. We show, both empirically and
theoretically, that these false negatives severely impair RL training by
depriving the model of informative gradient signals and slowing convergence. To
mitigate this, we propose tinyV, a lightweight LLM-based verifier that augments
existing rule-based methods, which dynamically identifies potential false
negatives and recovers valid responses to produce more accurate reward
estimates. Across multiple math-reasoning benchmarks, integrating TinyV boosts
pass rates by up to 10% and accelerates convergence relative to the baseline.
Our findings highlight the critical importance of addressing verifier false
negatives and offer a practical approach to improve RL-based fine-tuning of
LLMs. Our code is available at https://github.com/uw-nsl/TinyV.
TinyV, a lightweight LLM-based verifier, improves RL training of large language models by addressing false negatives from existing rule-based verifiers, enhancing reward accuracy and convergence speed.
Reinforcement Learning large language models policies reward signals verifiers false negatives Big-Math-RL-Verified dataset gradient signals convergence TinyV rule-based methods math-reasoning benchmarks pass rates