title: Trajectory Balance with Asynchrony: Decoupling Exploration and Learning
    for Fast, Scalable LLM Post-Training
summary: Reinforcement learning (RL) is a critical component of large language model
  (LLM) post-training. However, existing on-policy algorithms used for
  post-training are inherently incompatible with the use of experience replay
  buffers, which can be populated scalably by distributed off-policy actors to
  enhance exploration as compute increases. We propose efficiently obtaining this
  benefit of replay buffers via Trajectory Balance with Asynchrony (TBA), a
  massively scalable LLM RL system. In contrast to existing approaches, TBA uses
  a larger fraction of compute on search, constantly generating off-policy data
  for a central replay buffer. A training node simultaneously samples data from
  this buffer based on reward or recency to update the policy using Trajectory
  Balance (TB), a diversity-seeking RL objective introduced for GFlowNets. TBA
  offers three key advantages: (1) decoupled training and search, speeding up
  training wall-clock time by 4x or more; (2) improved diversity through
  large-scale off-policy sampling; and (3) scalable search for sparse reward
  settings. On mathematical reasoning, preference-tuning, and automated
  red-teaming (diverse and representative post-training tasks), TBA produces
  speed and performance improvements over strong baselines.
paper_authors._id: 67e5691d4240c0b10d6126ea
paper_authors.name: Brian R. Bartoldson
paper_authors._id: 67e5691d4240c0b10d6126eb
paper_authors.name: Siddarth Venkatraman
paper_authors._id: 67e5691d4240c0b10d6126ec
paper_authors.name: James Diffenderfer
paper_authors._id: 67e5691d4240c0b10d6126ed
paper_authors.name: Moksh Jain
paper_authors._id: 67e5691d4240c0b10d6126ee
paper_authors.name: Tal Ben-Nun
paper_authors._id: 67e5691d4240c0b10d6126ef
paper_authors.name: Seanie Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-20T07:23:17.489Z
paper_authors._id: 64ad5f59b7e4b2c1ce47eb43
paper_authors.avatarUrl: /avatars/1f13ebe21a90d8c99920aa2c8cd9ac45.svg
paper_authors.fullname: Seanie Lee
paper_authors.type: user
paper_authors.user: Seanie-lee
paper_authors._id: 67e5691d4240c0b10d6126f0
paper_authors.name: Minsu Kim
paper_authors._id: 67e5691d4240c0b10d6126f1
paper_authors.name: Johan Obando-Ceron
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-28T08:38:10.901Z
paper_authors._id: 6478f30ea68454566353ef95
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6478f30ea68454566353ef95/hX5NiVPyQbK8TIBKnT4J1.jpeg
paper_authors.fullname: Johan Samir Obando Ceron
paper_authors.type: user
paper_authors.user: johanobandoc
paper_authors._id: 67e5691d4240c0b10d6126f2
paper_authors.name: Yoshua Bengio
paper_authors._id: 67e5691d4240c0b10d6126f3
paper_authors.name: Bhavya Kailkhura
paper_title: Trajectory Balance with Asynchrony: Decoupling Exploration and Learning
    for Fast, Scalable LLM Post-Training
paper_summary: Reinforcement learning (RL) is a critical component of large language model
  (LLM) post-training. However, existing on-policy algorithms used for
  post-training are inherently incompatible with the use of experience replay
  buffers, which can be populated scalably by distributed off-policy actors to
  enhance exploration as compute increases. We propose efficiently obtaining this
  benefit of replay buffers via Trajectory Balance with Asynchrony (TBA), a
  massively scalable LLM RL system. In contrast to existing approaches, TBA uses
  a larger fraction of compute on search, constantly generating off-policy data
  for a central replay buffer. A training node simultaneously samples data from
  this buffer based on reward or recency to update the policy using Trajectory
  Balance (TB), a diversity-seeking RL objective introduced for GFlowNets. TBA
  offers three key advantages: (1) decoupled training and search, speeding up
  training wall-clock time by 4x or more; (2) improved diversity through
  large-scale off-policy sampling; and (3) scalable search for sparse reward
  settings. On mathematical reasoning, preference-tuning, and automated
  red-teaming (diverse and representative post-training tasks), TBA produces
  speed and performance improvements over strong baselines.
paper_ai_summary: Trajectory Balance with Asynchrony (TBA) decouples training and search in reinforcement learning for large language models, enhancing diversity and scalability with off-policy data.
paper_ai_keywords: reinforcement learning (RL) large language model (LLM) experience replay buffers Trajectory Balance with Asynchrony (TBA) off-policy sampling Trajectory Balance (TB) decoupled training sparse reward settings mathematical reasoning preference-tuning automated red-teaming