title: DUMP: Automated Distribution-Level Curriculum Learning for RL-based LLM
    Post-training
summary: Recent advances in reinforcement learning (RL)-based post-training have led
  to notable improvements in large language models (LLMs), particularly in
  enhancing their reasoning capabilities to handle complex tasks. However, most
  existing methods treat the training data as a unified whole, overlooking the
  fact that modern LLM training often involves a mixture of data from diverse
  distributions-varying in both source and difficulty. This heterogeneity
  introduces a key challenge: how to adaptively schedule training across
  distributions to optimize learning efficiency. In this paper, we present a
  principled curriculum learning framework grounded in the notion of
  distribution-level learnability. Our core insight is that the magnitude of
  policy advantages reflects how much a model can still benefit from further
  training on a given distribution. Based on this, we propose a
  distribution-level curriculum learning framework for RL-based LLM
  post-training, which leverages the Upper Confidence Bound (UCB) principle to
  dynamically adjust sampling probabilities for different distrubutions. This
  approach prioritizes distributions with either high average advantage
  (exploitation) or low sample count (exploration), yielding an adaptive and
  theoretically grounded training schedule. We instantiate our curriculum
  learning framework with GRPO as the underlying RL algorithm and demonstrate its
  effectiveness on logic reasoning datasets with multiple difficulties and
  sources. Our experiments show that our framework significantly improves
  convergence speed and final performance, highlighting the value of
  distribution-aware curriculum strategies in LLM post-training. Code:
  https://github.com/ZhentingWang/DUMP.
paper_authors._id: 67fdb42aa8deb632ed46d23d
paper_authors.name: Zhenting Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-15T07:54:59.070Z
paper_authors._id: 64dfcc62e8b6f3f3baa950e0
paper_authors.avatarUrl: /avatars/21bbff67d46c08044efe2406575aa77e.svg
paper_authors.fullname: Zhenting Wang
paper_authors.type: user
paper_authors.user: ztwang
paper_authors._id: 67fdb42aa8deb632ed46d23e
paper_authors.name: Guofeng Cui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-15T08:24:11.131Z
paper_authors._id: 6670b3b78eac2e222ebf77d4
paper_authors.avatarUrl: /avatars/0f1d231eac479ca78ddf106a72490faa.svg
paper_authors.fullname: Guofeng Cui
paper_authors.type: user
paper_authors.user: gfcui
paper_authors._id: 67fdb42aa8deb632ed46d23f
paper_authors.name: Kun Wan
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-04-15T01:22:46.116Z
paper_authors._id: 66274e02348a5304435dc9cc
paper_authors.avatarUrl: /avatars/bda87559cd497c310597c2fc8430b31f.svg
paper_authors.fullname: Kun Wan
paper_authors.type: user
paper_authors.user: timecuriosity
paper_authors._id: 67fdb42aa8deb632ed46d240
paper_authors.name: Wentian Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-15T08:24:30.442Z
paper_authors._id: 66443629b23fe8d3f7f2d0c7
paper_authors.avatarUrl: /avatars/98ff088036aa382f33a05c232604c565.svg
paper_authors.fullname: Wentian Zhao
paper_authors.type: user
paper_authors.user: zwt123home123
paper_title: DUMP: Automated Distribution-Level Curriculum Learning for RL-based LLM
    Post-training
paper_summary: Recent advances in reinforcement learning (RL)-based post-training have led
  to notable improvements in large language models (LLMs), particularly in
  enhancing their reasoning capabilities to handle complex tasks. However, most
  existing methods treat the training data as a unified whole, overlooking the
  fact that modern LLM training often involves a mixture of data from diverse
  distributions-varying in both source and difficulty. This heterogeneity
  introduces a key challenge: how to adaptively schedule training across
  distributions to optimize learning efficiency. In this paper, we present a
  principled curriculum learning framework grounded in the notion of
  distribution-level learnability. Our core insight is that the magnitude of
  policy advantages reflects how much a model can still benefit from further
  training on a given distribution. Based on this, we propose a
  distribution-level curriculum learning framework for RL-based LLM
  post-training, which leverages the Upper Confidence Bound (UCB) principle to
  dynamically adjust sampling probabilities for different distrubutions. This
  approach prioritizes distributions with either high average advantage
  (exploitation) or low sample count (exploration), yielding an adaptive and
  theoretically grounded training schedule. We instantiate our curriculum
  learning framework with GRPO as the underlying RL algorithm and demonstrate its
  effectiveness on logic reasoning datasets with multiple difficulties and
  sources. Our experiments show that our framework significantly improves
  convergence speed and final performance, highlighting the value of
  distribution-aware curriculum strategies in LLM post-training. Code:
  https://github.com/ZhentingWang/DUMP.
paper_ai_summary: A distribution-level curriculum learning framework using UCB for RL-based LLM post-training enhances convergence speed and performance by dynamically adjusting training schedules across diverse data distributions.
paper_ai_keywords: reinforcement learning large language models LLM curriculum learning distribution-level learnability Upper Confidence Bound UCB policy advantages convergence speed final performance distribution-aware curriculum strategies GRPO