UloRL:An Ultra-Long Output Reinforcement Learning Approach for Advancing
  Large Language Models' Reasoning Abilities
Recent advances in large language models (LLMs) have highlighted the
potential of reinforcement learning with verifiable rewards (RLVR) to enhance
reasoning capabilities through extended output sequences. However, traditional
RL frameworks face inefficiencies when handling ultra-long outputs due to
long-tail sequence distributions and entropy collapse during training. To
address these challenges, we propose an Ultra-Long Output Reinforcement
Learning (UloRL) approach for advancing large language models' reasoning
abilities. Specifically, we divide ultra long output decoding into short
segments, enabling efficient training by mitigating delays caused by long-tail
samples. Additionally, we introduce dynamic masking of well-Mastered Positive
Tokens (MPTs) to prevent entropy collapse. Experimental results demonstrate the
effectiveness of our approach. On the Qwen3-30B-A3B model, RL with segment
rollout achieved 2.06x increase in training speed, while RL training with
128k-token outputs improves the model's performance on AIME2025 from 70.9\% to
85.1\% and on BeyondAIME from 50.7\% to 61.9\%, even surpassing Qwen3-235B-A22B
with remarkable gains. These findings underscore the potential of our methods
to advance the reasoning capabilities of LLMs with ultra-long sequence
generation. We will release our code and model for further use by the
community.
6888299caf872d625c10c5b2
Dong Du
claimed_verified
2025-07-29T07:08:53.985Z
651eaaad567de56557e7bde6
/avatars/7ddf08fe4459fa2b7b3796be4eb4f43d.svg
duu
user
dongdongdongdu
6888299caf872d625c10c5b3
Shulin Liu
claimed_verified
2025-07-29T07:08:57.247Z
67cffe3d275d90f8097ae10a
/avatars/eb4d06abb06c677e4d1be77685fc23f6.svg
liu
user
forestliutc
6888299caf872d625c10c5b4
Tao Yang
6888299caf872d625c10c5b5
Shaohua Chen
6888299caf872d625c10c5b6
Yang Li
UloRL:An Ultra-Long Output Reinforcement Learning Approach for Advancing
  Large Language Models' Reasoning Abilities
Recent advances in large language models (LLMs) have highlighted the
potential of reinforcement learning with verifiable rewards (RLVR) to enhance
reasoning capabilities through extended output sequences. However, traditional
RL frameworks face inefficiencies when handling ultra-long outputs due to
long-tail sequence distributions and entropy collapse during training. To
address these challenges, we propose an Ultra-Long Output Reinforcement
Learning (UloRL) approach for advancing large language models' reasoning
abilities. Specifically, we divide ultra long output decoding into short
segments, enabling efficient training by mitigating delays caused by long-tail
samples. Additionally, we introduce dynamic masking of well-Mastered Positive
Tokens (MPTs) to prevent entropy collapse. Experimental results demonstrate the
effectiveness of our approach. On the Qwen3-30B-A3B model, RL with segment
rollout achieved 2.06x increase in training speed, while RL training with
128k-token outputs improves the model's performance on AIME2025 from 70.9\% to
85.1\% and on BeyondAIME from 50.7\% to 61.9\%, even surpassing Qwen3-235B-A22B
with remarkable gains. These findings underscore the potential of our methods
to advance the reasoning capabilities of LLMs with ultra-long sequence
generation. We will release our code and model for further use by the
community.
The Ultra-Long Output Reinforcement Learning (UloRL) approach enhances large language models' reasoning by segmenting long outputs and using dynamic masking, leading to improved performance and faster training.
reinforcement learning with verifiable rewards (RLVR) ultra-long outputs long-tail sequence distributions entropy collapse Ultra-Long Output Reinforcement Learning (UloRL) short segments dynamic masking well-Mastered Positive Tokens (MPTs) segment rollout Qwen3-30B-A3B AIME2025 BeyondAIME Qwen3-235B-A22B