title: Rethinking the Sampling Criteria in Reinforcement Learning for LLM
    Reasoning: A Competence-Difficulty Alignment Perspective
summary: Reinforcement learning exhibits potential in enhancing the reasoning
  abilities of large language models, yet it is hard to scale for the low sample
  efficiency during the rollout phase. Existing methods attempt to improve
  efficiency by scheduling problems based on problem difficulties. However, these
  approaches suffer from unstable and biased estimations of problem difficulty
  and fail to capture the alignment between model competence and problem
  difficulty in RL training, leading to suboptimal results. To tackle these
  limitations, this paper introduces Competence-Difficulty
  Alignment Sampling (CDAS), which enables accurate
  and stable estimation of problem difficulties by aggregating historical
  performance discrepancies of problems. Then the model competence is quantified
  to adaptively select problems whose difficulty is in alignment with the model's
  current competence using a fixed-point system. Experimental results across a
  range of challenging mathematical benchmarks show that CDAS achieves great
  improvements in both accuracy and efficiency. CDAS attains the highest average
  accuracy against baselines and exhibits significant speed advantages compared
  to Dynamic Sampling, a competitive strategy in DAPO, which is 2.33
  times slower than CDAS.
paper_authors._id: 6835264edf7cbb5c08ce28a5
paper_authors.name: Deyang Kong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T07:51:42.197Z
paper_authors._id: 65a0aade5fafc248c2156e95
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65a0aade5fafc248c2156e95/S9YjJMTuKc-U1cFizqUMA.jpeg
paper_authors.fullname: DeyangKong
paper_authors.type: user
paper_authors.user: DeyangKong
paper_authors._id: 6835264edf7cbb5c08ce28a6
paper_authors.name: Qi Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T12:52:37.239Z
paper_authors._id: 631578d5996c52bf014a016b
paper_authors.avatarUrl: /avatars/c08a51e546a48cf3ee8a3f58e4a67eda.svg
paper_authors.fullname: GuoQi
paper_authors.type: user
paper_authors.user: GuoQi
paper_authors._id: 6835264edf7cbb5c08ce28a7
paper_authors.name: Xiangyu Xi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T07:51:39.843Z
paper_authors._id: 63edb098679c2cc40abc6c2e
paper_authors.avatarUrl: /avatars/288c7229937c2c3f29fda6d17c7df2eb.svg
paper_authors.fullname: Xiangyu
paper_authors.type: user
paper_authors.user: xixy
paper_authors._id: 6835264edf7cbb5c08ce28a8
paper_authors.name: Wei Wang
paper_authors._id: 6835264edf7cbb5c08ce28a9
paper_authors.name: Jingang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T12:52:23.913Z
paper_authors._id: 647097cbcfd57849518e656b
paper_authors.avatarUrl: /avatars/c66fe0add29c1bde9e3a98bf4a8793b9.svg
paper_authors.fullname: Jingang Wang
paper_authors.type: user
paper_authors.user: bitwjg
paper_authors._id: 6835264edf7cbb5c08ce28aa
paper_authors.name: Xunliang Cai
paper_authors._id: 6835264edf7cbb5c08ce28ab
paper_authors.name: Shikun Zhang
paper_authors._id: 6835264edf7cbb5c08ce28ac
paper_authors.name: Wei Ye
paper_title: Rethinking the Sampling Criteria in Reinforcement Learning for LLM
    Reasoning: A Competence-Difficulty Alignment Perspective
paper_summary: Reinforcement learning exhibits potential in enhancing the reasoning
  abilities of large language models, yet it is hard to scale for the low sample
  efficiency during the rollout phase. Existing methods attempt to improve
  efficiency by scheduling problems based on problem difficulties. However, these
  approaches suffer from unstable and biased estimations of problem difficulty
  and fail to capture the alignment between model competence and problem
  difficulty in RL training, leading to suboptimal results. To tackle these
  limitations, this paper introduces Competence-Difficulty
  Alignment Sampling (CDAS), which enables accurate
  and stable estimation of problem difficulties by aggregating historical
  performance discrepancies of problems. Then the model competence is quantified
  to adaptively select problems whose difficulty is in alignment with the model's
  current competence using a fixed-point system. Experimental results across a
  range of challenging mathematical benchmarks show that CDAS achieves great
  improvements in both accuracy and efficiency. CDAS attains the highest average
  accuracy against baselines and exhibits significant speed advantages compared
  to Dynamic Sampling, a competitive strategy in DAPO, which is 2.33
  times slower than CDAS.
paper_ai_summary: CDAS addresses low sample efficiency in reinforcement learning by aligning model competence with problem difficulty, improving both accuracy and efficiency in mathematical benchmarks.
paper_ai_keywords: reinforcement learning competence-difficulty alignment sampling CDAS historical performance discrepancies fixed-point system dynamic sampling DAPO