The Invisible Leash: Why RLVR May Not Escape Its Origin
Recent advances in large reasoning models highlight Reinforcement Learning
with Verifiable Rewards (RLVR) as a promising method for enhancing AI's
capabilities, particularly in solving complex logical tasks. However, it
remains unclear whether RLVR truly expands a model's reasoning boundary or
merely amplifies high-reward outputs that the base model already knows for
improved precision. This study presents a theoretical and empirical
investigation that provides fresh insights into the potential limits of RLVR.
First, we offer a new theoretical perspective that RLVR is constrained by the
base model's support-unable to sample solutions with zero initial
probability-and operates as a conservative reweighting mechanism that may
restrict the discovery of entirely original solutions. We also identify an
entropy-reward tradeoff: while RLVR reliably enhances precision, it may
progressively narrow exploration and potentially overlook correct yet
underrepresented solutions. Extensive empirical experiments validate that while
RLVR consistently improves pass@1, the shrinkage of empirical support generally
outweighs the expansion of empirical support under larger sampling budgets,
failing to recover correct answers that were previously accessible to the base
model. Interestingly, we also observe that while RLVR sometimes increases
token-level entropy, resulting in greater uncertainty at each generation step,
answer-level entropy declines, indicating that these seemingly more uncertain
paths ultimately converge onto a smaller set of distinct answers. Taken
together, these findings reveal potential limits of RLVR in extending reasoning
horizons. Breaking this invisible leash may require future algorithmic
innovations such as explicit exploration mechanisms or hybrid strategies that
seed probability mass into underrepresented solution regions.
687f22ec33947f780d9b4b62
Fang Wu
claimed_verified
2025-07-23T08:37:15.483Z
675e0d5cdd3e9eeed6954f5a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/7oMEoBmaFiCR9K2q9Z_7q.png
Fang Wu
user
fangwu97
687f22ec33947f780d9b4b63
Weihao Xuan
claimed_verified
2025-07-22T07:47:39.731Z
65b8909c89eb3dfbe8d26780
https://cdn-avatars.huggingface.co/v1/production/uploads/65b8909c89eb3dfbe8d26780/aJExXsCN9oQOdd9bsQOEO.jpeg
Weihao XUAN
user
weihao1115
687f22ec33947f780d9b4b64
Ximing Lu
687f22ec33947f780d9b4b65
Zaid Harchaoui
687f22ec33947f780d9b4b66
Yejin Choi
The Invisible Leash: Why RLVR May Not Escape Its Origin
Recent advances in large reasoning models highlight Reinforcement Learning
with Verifiable Rewards (RLVR) as a promising method for enhancing AI's
capabilities, particularly in solving complex logical tasks. However, it
remains unclear whether RLVR truly expands a model's reasoning boundary or
merely amplifies high-reward outputs that the base model already knows for
improved precision. This study presents a theoretical and empirical
investigation that provides fresh insights into the potential limits of RLVR.
First, we offer a new theoretical perspective that RLVR is constrained by the
base model's support-unable to sample solutions with zero initial
probability-and operates as a conservative reweighting mechanism that may
restrict the discovery of entirely original solutions. We also identify an
entropy-reward tradeoff: while RLVR reliably enhances precision, it may
progressively narrow exploration and potentially overlook correct yet
underrepresented solutions. Extensive empirical experiments validate that while
RLVR consistently improves pass@1, the shrinkage of empirical support generally
outweighs the expansion of empirical support under larger sampling budgets,
failing to recover correct answers that were previously accessible to the base
model. Interestingly, we also observe that while RLVR sometimes increases
token-level entropy, resulting in greater uncertainty at each generation step,
answer-level entropy declines, indicating that these seemingly more uncertain
paths ultimately converge onto a smaller set of distinct answers. Taken
together, these findings reveal potential limits of RLVR in extending reasoning
horizons. Breaking this invisible leash may require future algorithmic
innovations such as explicit exploration mechanisms or hybrid strategies that
seed probability mass into underrepresented solution regions.
Reinforcement Learning with Verifiable Rewards (RLVR) enhances precision but may limit exploration and discovery of new solutions, suggesting potential limits to its effectiveness in expanding reasoning capabilities.
Reinforcement Learning with Verifiable Rewards RLVR support reweighting mechanism entropy-reward tradeoff pass@1 empirical support token-level entropy answer-level entropy exploration mechanisms hybrid strategies