title: AReaL: A Large-Scale Asynchronous Reinforcement Learning System for
    Language Reasoning
summary: Reinforcement learning (RL) has become a trending paradigm for training large
  language models (LLMs), particularly for reasoning tasks. Effective RL for LLMs
  requires massive parallelization and poses an urgent need for efficient
  training systems. Most existing large-scale RL systems for LLMs are synchronous
  by alternating generation and training in a batch setting, where the rollouts
  in each training batch are generated by the same (or latest) model. This
  stabilizes RL training but suffers from severe system-level inefficiency.
  Generation must wait until the longest output in the batch is completed before
  model update, resulting in GPU underutilization. We present AReaL, a
  fully asynchronous RL system that completely decouples generation from
  training. Rollout workers in AReaL continuously generate new outputs without
  waiting, while training workers update the model whenever a batch of data is
  collected. AReaL also incorporates a collection of system-level optimizations,
  leading to substantially higher GPU utilization. To stabilize RL training,
  AReaL balances the workload of rollout and training workers to control data
  staleness, and adopts a staleness-enhanced PPO variant to better handle
  outdated training samples. Extensive experiments on math and code reasoning
  benchmarks show that AReaL achieves up to 2.57times training
  speedup compared to the best synchronous systems with the same number of GPUs
  and matched or even improved final performance. The code of AReaL is available
  at https://github.com/inclusionAI/AReaL/.
paper_authors._id: 683d12963aa5ac98190e1eda
paper_authors.name: Wei Fu
paper_authors._id: 683d12963aa5ac98190e1edb
paper_authors.name: Jiaxuan Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T12:39:56.082Z
paper_authors._id: 65338f12392f3bcb323d78b2
paper_authors.avatarUrl: /avatars/960b9338298ff6c663a6706aa6189e72.svg
paper_authors.fullname: Jiaxuan Gao
paper_authors.type: user
paper_authors.user: samjia2000
paper_authors._id: 683d12963aa5ac98190e1edc
paper_authors.name: Xujie Shen
paper_authors._id: 683d12963aa5ac98190e1edd
paper_authors.name: Chen Zhu
paper_authors._id: 683d12963aa5ac98190e1ede
paper_authors.name: Zhiyu Mei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T12:40:20.790Z
paper_authors._id: 66d782669b7da501cd079628
paper_authors.avatarUrl: /avatars/5fc76cc6d11588c4d2404989ca400809.svg
paper_authors.fullname: Zhi Yu mei
paper_authors.type: user
paper_authors.user: Souleaf
paper_authors._id: 683d12963aa5ac98190e1edf
paper_authors.name: Chuyi He
paper_authors._id: 683d12963aa5ac98190e1ee0
paper_authors.name: Shusheng Xu
paper_authors._id: 683d12963aa5ac98190e1ee1
paper_authors.name: Guo Wei
paper_authors._id: 683d12963aa5ac98190e1ee2
paper_authors.name: Jun Mei
paper_authors._id: 683d12963aa5ac98190e1ee3
paper_authors.name: Jiashu Wang
paper_authors._id: 683d12963aa5ac98190e1ee4
paper_authors.name: Tongkai Yang
paper_authors._id: 683d12963aa5ac98190e1ee5
paper_authors.name: Binhang Yuan
paper_authors._id: 683d12963aa5ac98190e1ee6
paper_authors.name: Yi Wu
paper_title: AReaL: A Large-Scale Asynchronous Reinforcement Learning System for
    Language Reasoning
paper_summary: Reinforcement learning (RL) has become a trending paradigm for training large
  language models (LLMs), particularly for reasoning tasks. Effective RL for LLMs
  requires massive parallelization and poses an urgent need for efficient
  training systems. Most existing large-scale RL systems for LLMs are synchronous
  by alternating generation and training in a batch setting, where the rollouts
  in each training batch are generated by the same (or latest) model. This
  stabilizes RL training but suffers from severe system-level inefficiency.
  Generation must wait until the longest output in the batch is completed before
  model update, resulting in GPU underutilization. We present AReaL, a
  fully asynchronous RL system that completely decouples generation from
  training. Rollout workers in AReaL continuously generate new outputs without
  waiting, while training workers update the model whenever a batch of data is
  collected. AReaL also incorporates a collection of system-level optimizations,
  leading to substantially higher GPU utilization. To stabilize RL training,
  AReaL balances the workload of rollout and training workers to control data
  staleness, and adopts a staleness-enhanced PPO variant to better handle
  outdated training samples. Extensive experiments on math and code reasoning
  benchmarks show that AReaL achieves up to 2.57times training
  speedup compared to the best synchronous systems with the same number of GPUs
  and matched or even improved final performance. The code of AReaL is available
  at https://github.com/inclusionAI/AReaL/.
paper_ai_summary: AReaL, a fully asynchronous reinforcement learning system, decouples generation and training to achieve higher GPU utilization and up to 2.57x training speedup for large language models on reasoning tasks.
paper_ai_keywords: reinforcement learning large language models asynchronous system rollouts model update GPU utilization PPO data staleness training speedup