Rethinking the Sampling Criteria in Reinforcement Learning for LLM
  Reasoning: A Competence-Difficulty Alignment Perspective
Reinforcement learning exhibits potential in enhancing the reasoning
abilities of large language models, yet it is hard to scale for the low sample
efficiency during the rollout phase. Existing methods attempt to improve
efficiency by scheduling problems based on problem difficulties. However, these
approaches suffer from unstable and biased estimations of problem difficulty
and fail to capture the alignment between model competence and problem
difficulty in RL training, leading to suboptimal results. To tackle these
limitations, this paper introduces Competence-Difficulty
Alignment Sampling (CDAS), which enables accurate
and stable estimation of problem difficulties by aggregating historical
performance discrepancies of problems. Then the model competence is quantified
to adaptively select problems whose difficulty is in alignment with the model's
current competence using a fixed-point system. Experimental results across a
range of challenging mathematical benchmarks show that CDAS achieves great
improvements in both accuracy and efficiency. CDAS attains the highest average
accuracy against baselines and exhibits significant speed advantages compared
to Dynamic Sampling, a competitive strategy in DAPO, which is 2.33
times slower than CDAS.
6835264edf7cbb5c08ce28a5
Deyang Kong
claimed_verified
2025-05-27T07:51:42.197Z
65a0aade5fafc248c2156e95
https://cdn-avatars.huggingface.co/v1/production/uploads/65a0aade5fafc248c2156e95/S9YjJMTuKc-U1cFizqUMA.jpeg
DeyangKong
user
DeyangKong
6835264edf7cbb5c08ce28a6
Qi Guo
admin_assigned
2025-05-27T12:52:37.239Z
631578d5996c52bf014a016b
/avatars/c08a51e546a48cf3ee8a3f58e4a67eda.svg
GuoQi
user
GuoQi
6835264edf7cbb5c08ce28a7
Xiangyu Xi
claimed_verified
2025-05-27T07:51:39.843Z
63edb098679c2cc40abc6c2e
/avatars/288c7229937c2c3f29fda6d17c7df2eb.svg
Xiangyu
user
xixy
6835264edf7cbb5c08ce28a8
Wei Wang
6835264edf7cbb5c08ce28a9
Jingang Wang
admin_assigned
2025-05-27T12:52:23.913Z
647097cbcfd57849518e656b
/avatars/c66fe0add29c1bde9e3a98bf4a8793b9.svg
Jingang Wang
user
bitwjg
6835264edf7cbb5c08ce28aa
Xunliang Cai
6835264edf7cbb5c08ce28ab
Shikun Zhang
6835264edf7cbb5c08ce28ac
Wei Ye
Rethinking the Sampling Criteria in Reinforcement Learning for LLM
  Reasoning: A Competence-Difficulty Alignment Perspective
Reinforcement learning exhibits potential in enhancing the reasoning
abilities of large language models, yet it is hard to scale for the low sample
efficiency during the rollout phase. Existing methods attempt to improve
efficiency by scheduling problems based on problem difficulties. However, these
approaches suffer from unstable and biased estimations of problem difficulty
and fail to capture the alignment between model competence and problem
difficulty in RL training, leading to suboptimal results. To tackle these
limitations, this paper introduces Competence-Difficulty
Alignment Sampling (CDAS), which enables accurate
and stable estimation of problem difficulties by aggregating historical
performance discrepancies of problems. Then the model competence is quantified
to adaptively select problems whose difficulty is in alignment with the model's
current competence using a fixed-point system. Experimental results across a
range of challenging mathematical benchmarks show that CDAS achieves great
improvements in both accuracy and efficiency. CDAS attains the highest average
accuracy against baselines and exhibits significant speed advantages compared
to Dynamic Sampling, a competitive strategy in DAPO, which is 2.33
times slower than CDAS.
CDAS addresses low sample efficiency in reinforcement learning by aligning model competence with problem difficulty, improving both accuracy and efficiency in mathematical benchmarks.
reinforcement learning competence-difficulty alignment sampling CDAS historical performance discrepancies fixed-point system dynamic sampling DAPO