Efficient Reinforcement Finetuning via Adaptive Curriculum Learning
Reinforcement finetuning (RFT) has shown great potential for enhancing the
mathematical reasoning capabilities of large language models (LLMs), but it is
often sample- and compute-inefficient, requiring extensive training. In this
work, we introduce AdaRFT (Adaptive Curriculum Reinforcement Finetuning), a
method that significantly improves both the efficiency and final accuracy of
RFT through adaptive curriculum learning. AdaRFT dynamically adjusts the
difficulty of training problems based on the model's recent reward signals,
ensuring that the model consistently trains on tasks that are challenging but
solvable. This adaptive sampling strategy accelerates learning by maintaining
an optimal difficulty range, avoiding wasted computation on problems that are
too easy or too hard. AdaRFT requires only a lightweight extension to standard
RFT algorithms like Proximal Policy Optimization (PPO), without modifying the
reward function or model architecture. Experiments on competition-level math
datasets-including AMC, AIME, and IMO-style problems-demonstrate that AdaRFT
significantly improves both training efficiency and reasoning performance. We
evaluate AdaRFT across multiple data distributions and model sizes, showing
that it reduces the number of training steps by up to 2x and improves accuracy
by a considerable margin, offering a more scalable and effective RFT framework.
67f5e81488885c9950c7ab7e
Taiwei Shi
claimed_verified
2025-04-09T07:34:46.737Z
62e1b3cb3eb0730f621a83f6
https://cdn-avatars.huggingface.co/v1/production/uploads/1658958764563-noauth.jpeg
Taiwei Shi
user
MaksimSTW
67f5e81488885c9950c7ab7f
Yiyang Wu
67f5e81488885c9950c7ab80
Linxin Song
claimed_verified
2025-04-14T09:47:18.915Z
64d660308ebc40443813f014
/avatars/516bb2d2383be99794e366dfb41636b6.svg
Linxin Song
user
linxinso
67f5e81488885c9950c7ab81
Tianyi Zhou
claimed_verified
2025-04-16T09:24:20.414Z
647f5af5b0e96764589f3b2a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
Tianyi Zhou
user
zhoutianyi
67f5e81488885c9950c7ab82
Jieyu Zhao
Efficient Reinforcement Finetuning via Adaptive Curriculum Learning
Reinforcement finetuning (RFT) has shown great potential for enhancing the
mathematical reasoning capabilities of large language models (LLMs), but it is
often sample- and compute-inefficient, requiring extensive training. In this
work, we introduce AdaRFT (Adaptive Curriculum Reinforcement Finetuning), a
method that significantly improves both the efficiency and final accuracy of
RFT through adaptive curriculum learning. AdaRFT dynamically adjusts the
difficulty of training problems based on the model's recent reward signals,
ensuring that the model consistently trains on tasks that are challenging but
solvable. This adaptive sampling strategy accelerates learning by maintaining
an optimal difficulty range, avoiding wasted computation on problems that are
too easy or too hard. AdaRFT requires only a lightweight extension to standard
RFT algorithms like Proximal Policy Optimization (PPO), without modifying the
reward function or model architecture. Experiments on competition-level math
datasets-including AMC, AIME, and IMO-style problems-demonstrate that AdaRFT
significantly improves both training efficiency and reasoning performance. We
evaluate AdaRFT across multiple data distributions and model sizes, showing
that it reduces the number of training steps by up to 2x and improves accuracy
by a considerable margin, offering a more scalable and effective RFT framework.
AdaRFT uses adaptive curriculum learning to improve the efficiency and accuracy of reinforcement fine-tuning for large language models in mathematical reasoning.
Reinforcement finetuning (RFT) AdaRFT adaptive curriculum learning reward signals Proximal Policy Optimization (PPO) AMC AIME IMO training efficiency reasoning performance