Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language
  Models
The dominant paradigm for RLHF is online and on-policy RL: synchronously
generating from the large language model (LLM) policy, labelling with a reward
model, and learning using feedback on the LLM's own outputs. While performant,
this paradigm is computationally inefficient. Inspired by classical deep RL
literature, we propose separating generation and learning in RLHF. This enables
asynchronous generation of new samples while simultaneously training on old
samples, leading to faster training and more compute-optimal scaling. However,
asynchronous training relies on an underexplored regime, online but off-policy
RLHF: learning on samples from previous iterations of our model. To understand
the challenges in this regime, we investigate a fundamental question: how much
off-policyness can we tolerate for asynchronous training to speed up learning
but maintain performance? Among several RLHF algorithms we tested, we find that
online DPO is most robust to off-policy data, and robustness increases with the
scale of the policy model. We study further compute optimizations for
asynchronous RLHF but find that they come at a performance cost, giving rise to
a trade-off. Finally, we verify the scalability of asynchronous RLHF by
training LLaMA 3.1 8B on an instruction-following task 40% faster than a
synchronous run while matching final performance.
671bdeab8411a002370ae6e0
Michael Noukhovitch
extracted_pending
2024-10-25T18:08:44.732Z
62fc4d3ae44837de544a5919
/avatars/65dc6943b0da835d3bfe9c1d4d7a9f76.svg
Michael N
user
mnoukhov
671bdeab8411a002370ae6e1
Shengyi Huang
claimed_verified
2025-01-05T23:03:26.869Z
62bfc80b54db40ca617ce53a
https://cdn-avatars.huggingface.co/v1/production/uploads/62bfc80b54db40ca617ce53a/IZxafMqoYJNHwTuFt8AqF.png
Shengyi Costa Huang
user
vwxyzjn
671bdeab8411a002370ae6e2
Sophie Xhonneux
671bdeab8411a002370ae6e3
Arian Hosseini
671bdeab8411a002370ae6e4
Rishabh Agarwal
671bdeab8411a002370ae6e5
Aaron Courville
Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language
  Models
The dominant paradigm for RLHF is online and on-policy RL: synchronously
generating from the large language model (LLM) policy, labelling with a reward
model, and learning using feedback on the LLM's own outputs. While performant,
this paradigm is computationally inefficient. Inspired by classical deep RL
literature, we propose separating generation and learning in RLHF. This enables
asynchronous generation of new samples while simultaneously training on old
samples, leading to faster training and more compute-optimal scaling. However,
asynchronous training relies on an underexplored regime, online but off-policy
RLHF: learning on samples from previous iterations of our model. To understand
the challenges in this regime, we investigate a fundamental question: how much
off-policyness can we tolerate for asynchronous training to speed up learning
but maintain performance? Among several RLHF algorithms we tested, we find that
online DPO is most robust to off-policy data, and robustness increases with the
scale of the policy model. We study further compute optimizations for
asynchronous RLHF but find that they come at a performance cost, giving rise to
a trade-off. Finally, we verify the scalability of asynchronous RLHF by
training LLaMA 3.1 8B on an instruction-following task 40% faster than a
synchronous run while matching final performance.
Asynchronous generation and learning in Reinforcement Learning from Human Feedback (RLHF) improve training speed and computational efficiency while maintaining performance, particularly with larger policy models.
RLHF online RL on-policy RL large language model reward model asynchronous generation off-policy RLHF online DPO instruction-following task scalability