Reinforcement Learning for Reasoning in Large Language Models with One
  Training Example
We show that reinforcement learning with verifiable reward using one training
example (1-shot RLVR) is effective in incentivizing the math reasoning
capabilities of large language models (LLMs). Applying RLVR to the base model
Qwen2.5-Math-1.5B, we identify a single example that elevates model performance
on MATH500 from 36.0% to 73.6%, and improves the average performance across six
common mathematical reasoning benchmarks from 17.6% to 35.7%. This result
matches the performance obtained using the 1.2k DeepScaleR subset (MATH500:
73.6%, average: 35.9%), which includes the aforementioned example. Similar
substantial improvements are observed across various models (Qwen2.5-Math-7B,
Llama3.2-3B-Instruct, DeepSeek-R1-Distill-Qwen-1.5B), RL algorithms (GRPO and
PPO), and different math examples (many of which yield approximately 30% or
greater improvement on MATH500 when employed as a single training example). In
addition, we identify some interesting phenomena during 1-shot RLVR, including
cross-domain generalization, increased frequency of self-reflection, and
sustained test performance improvement even after the training accuracy has
saturated, a phenomenon we term post-saturation generalization. Moreover, we
verify that the effectiveness of 1-shot RLVR primarily arises from the policy
gradient loss, distinguishing it from the "grokking" phenomenon. We also show
the critical role of promoting exploration (e.g., by adding entropy loss with
an appropriate coefficient) in 1-shot RLVR training. As a bonus, we observe
that applying entropy loss alone, without any outcome reward, significantly
enhances Qwen2.5-Math-1.5B's performance on MATH500 by 27.4%. These findings
can inspire future work on RLVR data efficiency and encourage a re-examination
of both recent progress and the underlying mechanisms in RLVR. Our code, model,
and data are open source at https://github.com/ypwang61/One-Shot-RLVR
681187ddda5ce4cbd7556714
Yiping Wang
admin_assigned
2025-04-30T09:58:59.486Z
653586fae778506c5b38a3f1
https://cdn-avatars.huggingface.co/v1/production/uploads/653586fae778506c5b38a3f1/GL_RShZhAkEZmIinA5_8E.jpeg
Yiping Wang
user
ypwang61
681187ddda5ce4cbd7556715
Qing Yang
claimed_verified
2025-04-30T07:56:17.953Z
673a83b99e6f1c0d81a771fc
/avatars/f3d8e1bf7d4c36b21adee632ea12ffe0.svg
Qing Yang
user
hushqyang
681187ddda5ce4cbd7556716
Zhiyuan Zeng
admin_assigned
2025-04-30T09:59:12.620Z
64a85e23b6512b8328f9d9e2
/avatars/4a6b35752d3f76cb03278f52b3b43426.svg
Zhiyuan Zeng
user
ZhiyuanZeng
681187ddda5ce4cbd7556717
Liliang Ren
admin_assigned
2025-04-30T09:59:18.310Z
63815eff4761ddfa00903762
https://cdn-avatars.huggingface.co/v1/production/uploads/63815eff4761ddfa00903762/YlkfX3sTZRQcoapctXJeJ.jpeg
Liliang Ren
user
renll
681187ddda5ce4cbd7556718
Lucas Liu
681187ddda5ce4cbd7556719
Baolin Peng
admin_assigned
2025-04-30T09:59:45.735Z
61942296d5c2ba6daa290357
/avatars/594021cc183c4922d48b46f43772a062.svg
Baolin Peng
user
Baolin
681187ddda5ce4cbd755671a
Hao Cheng
681187ddda5ce4cbd755671b
Xuehai He
admin_assigned
2025-04-30T09:59:52.344Z
6310493158d83e8f64dc8c55
/avatars/5f91ac4dfec0d6a5bf7bad6094f0fd0f.svg
Xuehai He
user
Xuehai
681187ddda5ce4cbd755671c
Kuan Wang
admin_assigned
2025-04-30T09:59:58.392Z
633523b131a2be3938ca1016
/avatars/06a18f80927289bb949d9f19ffdc4bda.svg
Kuan Wang
user
Keynes
681187ddda5ce4cbd755671d
Jianfeng Gao
admin_assigned
2025-04-30T10:00:04.685Z
641904caf9d6f1d772ec7af7
/avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
Jianfeng Gao
user
wyngjf
681187ddda5ce4cbd755671e
Weizhu Chen
admin_assigned
2025-04-30T10:00:10.823Z
64da876370446182be5b608d
/avatars/e412fdc71404ecdf638e416846e3ebfb.svg
Weizhu Chen
user
chenweizhu
681187ddda5ce4cbd755671f
Shuohang Wang
admin_assigned
2025-04-30T10:00:19.855Z
6463b2247572c66a8e625a57
/avatars/7722fb5649d42d966ce1e478946d5f8f.svg
Wang
user
Shuohang
681187ddda5ce4cbd7556720
Simon Shaolei Du
681187ddda5ce4cbd7556721
Yelong Shen
admin_assigned
2025-04-30T10:00:33.179Z
6454c337a13edf669cd5d8ea
/avatars/a383a0dda7c2ef6a0d6c3c64651f42ff.svg
Yelong Shen
user
uuu6
Reinforcement Learning for Reasoning in Large Language Models with One
  Training Example
We show that reinforcement learning with verifiable reward using one training
example (1-shot RLVR) is effective in incentivizing the math reasoning
capabilities of large language models (LLMs). Applying RLVR to the base model
Qwen2.5-Math-1.5B, we identify a single example that elevates model performance
on MATH500 from 36.0% to 73.6%, and improves the average performance across six
common mathematical reasoning benchmarks from 17.6% to 35.7%. This result
matches the performance obtained using the 1.2k DeepScaleR subset (MATH500:
73.6%, average: 35.9%), which includes the aforementioned example. Similar
substantial improvements are observed across various models (Qwen2.5-Math-7B,
Llama3.2-3B-Instruct, DeepSeek-R1-Distill-Qwen-1.5B), RL algorithms (GRPO and
PPO), and different math examples (many of which yield approximately 30% or
greater improvement on MATH500 when employed as a single training example). In
addition, we identify some interesting phenomena during 1-shot RLVR, including
cross-domain generalization, increased frequency of self-reflection, and
sustained test performance improvement even after the training accuracy has
saturated, a phenomenon we term post-saturation generalization. Moreover, we
verify that the effectiveness of 1-shot RLVR primarily arises from the policy
gradient loss, distinguishing it from the "grokking" phenomenon. We also show
the critical role of promoting exploration (e.g., by adding entropy loss with
an appropriate coefficient) in 1-shot RLVR training. As a bonus, we observe
that applying entropy loss alone, without any outcome reward, significantly
enhances Qwen2.5-Math-1.5B's performance on MATH500 by 27.4%. These findings
can inspire future work on RLVR data efficiency and encourage a re-examination
of both recent progress and the underlying mechanisms in RLVR. Our code, model,
and data are open source at https://github.com/ypwang61/One-Shot-RLVR
Reinforcement learning with verifiable reward using one training example significantly enhances math reasoning capabilities of large language models.
reinforcement learning verifiable reward 1-shot RLVR large language models Qwen2.5-Math-1.5B MATH500 mathematical reasoning benchmarks Qwen2.5-Math-7B Llama3.2-3B-Instruct DeepSeek-R1-Distill-Qwen-1.5B GRPO PPO policy gradient loss entropy loss cross-domain generalization self-reflection post-saturation generalization grokking