DUMP: Automated Distribution-Level Curriculum Learning for RL-based LLM
  Post-training
Recent advances in reinforcement learning (RL)-based post-training have led
to notable improvements in large language models (LLMs), particularly in
enhancing their reasoning capabilities to handle complex tasks. However, most
existing methods treat the training data as a unified whole, overlooking the
fact that modern LLM training often involves a mixture of data from diverse
distributions-varying in both source and difficulty. This heterogeneity
introduces a key challenge: how to adaptively schedule training across
distributions to optimize learning efficiency. In this paper, we present a
principled curriculum learning framework grounded in the notion of
distribution-level learnability. Our core insight is that the magnitude of
policy advantages reflects how much a model can still benefit from further
training on a given distribution. Based on this, we propose a
distribution-level curriculum learning framework for RL-based LLM
post-training, which leverages the Upper Confidence Bound (UCB) principle to
dynamically adjust sampling probabilities for different distrubutions. This
approach prioritizes distributions with either high average advantage
(exploitation) or low sample count (exploration), yielding an adaptive and
theoretically grounded training schedule. We instantiate our curriculum
learning framework with GRPO as the underlying RL algorithm and demonstrate its
effectiveness on logic reasoning datasets with multiple difficulties and
sources. Our experiments show that our framework significantly improves
convergence speed and final performance, highlighting the value of
distribution-aware curriculum strategies in LLM post-training. Code:
https://github.com/ZhentingWang/DUMP.
67fdb42aa8deb632ed46d23d
Zhenting Wang
claimed_verified
2025-04-15T07:54:59.070Z
64dfcc62e8b6f3f3baa950e0
/avatars/21bbff67d46c08044efe2406575aa77e.svg
Zhenting Wang
user
ztwang
67fdb42aa8deb632ed46d23e
Guofeng Cui
admin_assigned
2025-04-15T08:24:11.131Z
6670b3b78eac2e222ebf77d4
/avatars/0f1d231eac479ca78ddf106a72490faa.svg
Guofeng Cui
user
gfcui
67fdb42aa8deb632ed46d23f
Kun Wan
extracted_confirmed
2025-04-15T01:22:46.116Z
66274e02348a5304435dc9cc
/avatars/bda87559cd497c310597c2fc8430b31f.svg
Kun Wan
user
timecuriosity
67fdb42aa8deb632ed46d240
Wentian Zhao
admin_assigned
2025-04-15T08:24:30.442Z
66443629b23fe8d3f7f2d0c7
/avatars/98ff088036aa382f33a05c232604c565.svg
Wentian Zhao
user
zwt123home123
DUMP: Automated Distribution-Level Curriculum Learning for RL-based LLM
  Post-training
Recent advances in reinforcement learning (RL)-based post-training have led
to notable improvements in large language models (LLMs), particularly in
enhancing their reasoning capabilities to handle complex tasks. However, most
existing methods treat the training data as a unified whole, overlooking the
fact that modern LLM training often involves a mixture of data from diverse
distributions-varying in both source and difficulty. This heterogeneity
introduces a key challenge: how to adaptively schedule training across
distributions to optimize learning efficiency. In this paper, we present a
principled curriculum learning framework grounded in the notion of
distribution-level learnability. Our core insight is that the magnitude of
policy advantages reflects how much a model can still benefit from further
training on a given distribution. Based on this, we propose a
distribution-level curriculum learning framework for RL-based LLM
post-training, which leverages the Upper Confidence Bound (UCB) principle to
dynamically adjust sampling probabilities for different distrubutions. This
approach prioritizes distributions with either high average advantage
(exploitation) or low sample count (exploration), yielding an adaptive and
theoretically grounded training schedule. We instantiate our curriculum
learning framework with GRPO as the underlying RL algorithm and demonstrate its
effectiveness on logic reasoning datasets with multiple difficulties and
sources. Our experiments show that our framework significantly improves
convergence speed and final performance, highlighting the value of
distribution-aware curriculum strategies in LLM post-training. Code:
https://github.com/ZhentingWang/DUMP.
A distribution-level curriculum learning framework using UCB for RL-based LLM post-training enhances convergence speed and performance by dynamically adjusting training schedules across diverse data distributions.
reinforcement learning large language models LLM curriculum learning distribution-level learnability Upper Confidence Bound UCB policy advantages convergence speed final performance distribution-aware curriculum strategies GRPO