title: Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language
    Models
summary: The dominant paradigm for RLHF is online and on-policy RL: synchronously
  generating from the large language model (LLM) policy, labelling with a reward
  model, and learning using feedback on the LLM's own outputs. While performant,
  this paradigm is computationally inefficient. Inspired by classical deep RL
  literature, we propose separating generation and learning in RLHF. This enables
  asynchronous generation of new samples while simultaneously training on old
  samples, leading to faster training and more compute-optimal scaling. However,
  asynchronous training relies on an underexplored regime, online but off-policy
  RLHF: learning on samples from previous iterations of our model. To understand
  the challenges in this regime, we investigate a fundamental question: how much
  off-policyness can we tolerate for asynchronous training to speed up learning
  but maintain performance? Among several RLHF algorithms we tested, we find that
  online DPO is most robust to off-policy data, and robustness increases with the
  scale of the policy model. We study further compute optimizations for
  asynchronous RLHF but find that they come at a performance cost, giving rise to
  a trade-off. Finally, we verify the scalability of asynchronous RLHF by
  training LLaMA 3.1 8B on an instruction-following task 40% faster than a
  synchronous run while matching final performance.
paper_authors._id: 671bdeab8411a002370ae6e0
paper_authors.name: Michael Noukhovitch
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-10-25T18:08:44.732Z
paper_authors._id: 62fc4d3ae44837de544a5919
paper_authors.avatarUrl: /avatars/65dc6943b0da835d3bfe9c1d4d7a9f76.svg
paper_authors.fullname: Michael N
paper_authors.type: user
paper_authors.user: mnoukhov
paper_authors._id: 671bdeab8411a002370ae6e1
paper_authors.name: Shengyi Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-05T23:03:26.869Z
paper_authors._id: 62bfc80b54db40ca617ce53a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62bfc80b54db40ca617ce53a/IZxafMqoYJNHwTuFt8AqF.png
paper_authors.fullname: Shengyi Costa Huang
paper_authors.type: user
paper_authors.user: vwxyzjn
paper_authors._id: 671bdeab8411a002370ae6e2
paper_authors.name: Sophie Xhonneux
paper_authors._id: 671bdeab8411a002370ae6e3
paper_authors.name: Arian Hosseini
paper_authors._id: 671bdeab8411a002370ae6e4
paper_authors.name: Rishabh Agarwal
paper_authors._id: 671bdeab8411a002370ae6e5
paper_authors.name: Aaron Courville
paper_title: Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language
    Models
paper_summary: The dominant paradigm for RLHF is online and on-policy RL: synchronously
  generating from the large language model (LLM) policy, labelling with a reward
  model, and learning using feedback on the LLM's own outputs. While performant,
  this paradigm is computationally inefficient. Inspired by classical deep RL
  literature, we propose separating generation and learning in RLHF. This enables
  asynchronous generation of new samples while simultaneously training on old
  samples, leading to faster training and more compute-optimal scaling. However,
  asynchronous training relies on an underexplored regime, online but off-policy
  RLHF: learning on samples from previous iterations of our model. To understand
  the challenges in this regime, we investigate a fundamental question: how much
  off-policyness can we tolerate for asynchronous training to speed up learning
  but maintain performance? Among several RLHF algorithms we tested, we find that
  online DPO is most robust to off-policy data, and robustness increases with the
  scale of the policy model. We study further compute optimizations for
  asynchronous RLHF but find that they come at a performance cost, giving rise to
  a trade-off. Finally, we verify the scalability of asynchronous RLHF by
  training LLaMA 3.1 8B on an instruction-following task 40% faster than a
  synchronous run while matching final performance.
paper_ai_summary: Asynchronous generation and learning in Reinforcement Learning from Human Feedback (RLHF) improve training speed and computational efficiency while maintaining performance, particularly with larger policy models.
paper_ai_keywords: RLHF online RL on-policy RL large language model reward model asynchronous generation off-policy RLHF online DPO instruction-following task scalability