title: OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework
summary: As large language models (LLMs) continue to grow by scaling laws,
  reinforcement learning from human feedback (RLHF) has gained significant
  attention due to its outstanding performance. However, unlike pretraining or
  fine-tuning a single model, scaling reinforcement learning from human feedback
  (RLHF) for training large language models poses coordination challenges across
  four models. We present OpenRLHF, an open-source framework enabling efficient
  RLHF scaling. Unlike existing RLHF frameworks that co-locate four models on the
  same GPUs, OpenRLHF re-designs scheduling for the models beyond 70B parameters
  using Ray, vLLM, and DeepSpeed, leveraging improved resource utilization and
  diverse training approaches. Integrating seamlessly with Hugging Face, OpenRLHF
  provides an out-of-the-box solution with optimized algorithms and launch
  scripts, which ensures user-friendliness. OpenRLHF implements RLHF, DPO,
  rejection sampling, and other alignment techniques. Empowering state-of-the-art
  LLM development, OpenRLHF's code is available at
  https://github.com/OpenLLMAI/OpenRLHF.
paper_authors._id: 664c0b558e6575f45e13d2ad
paper_authors.name: Jian Hu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-06-05T00:20:51.693Z
paper_authors._id: 63f6c04ac96958470d1e9043
paper_authors.avatarUrl: /avatars/da46cdd9e21498e120ca91b67bfbfb5e.svg
paper_authors.fullname: Jian Hu
paper_authors.type: user
paper_authors.user: chuyi777
paper_authors._id: 664c0b558e6575f45e13d2ae
paper_authors.name: Xibin Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T08:13:06.288Z
paper_authors._id: 64f5a6b46e335217131d7d99
paper_authors.avatarUrl: /avatars/c4a15eadc5e612947a314a4a66a51c75.svg
paper_authors.fullname: Xibin Wu
paper_authors.type: user
paper_authors.user: wuxibin
paper_authors._id: 664c0b558e6575f45e13d2af
paper_authors.name: Weixun Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-21T08:59:18.059Z
paper_authors._id: 6263541963f73be3d2f91bdd
paper_authors.avatarUrl: /avatars/57474711c7e7768169509eb70e47176e.svg
paper_authors.fullname: weixun
paper_authors.type: user
paper_authors.user: weixun
paper_authors._id: 664c0b558e6575f45e13d2b0
paper_authors.name: Xianyu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-21T14:15:45.678Z
paper_authors._id: 6054383110aba34e3b6a2f02
paper_authors.avatarUrl: /avatars/63ff3a2ce469da0f59e9ef501250a136.svg
paper_authors.fullname: Xianyu
paper_authors.type: user
paper_authors.user: catqaq
paper_authors._id: 664c0b558e6575f45e13d2b1
paper_authors.name: Dehao Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T08:14:28.286Z
paper_authors._id: 634d0eeffc4873ed0d00406a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665994473058-noauth.jpeg
paper_authors.fullname: ZhangDehao
paper_authors.type: user
paper_authors.user: ispoon
paper_authors._id: 664c0b558e6575f45e13d2b2
paper_authors.name: Yu Cao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-27T07:20:12.327Z
paper_authors._id: 64618ad756c57120a2d1990c
paper_authors.avatarUrl: /avatars/1339e57e4d314116a72191d2d2d88f2c.svg
paper_authors.fullname: Tony Cao
paper_authors.type: user
paper_authors.user: Tony991
paper_title: OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework
paper_summary: As large language models (LLMs) continue to grow by scaling laws,
  reinforcement learning from human feedback (RLHF) has gained significant
  attention due to its outstanding performance. However, unlike pretraining or
  fine-tuning a single model, scaling reinforcement learning from human feedback
  (RLHF) for training large language models poses coordination challenges across
  four models. We present OpenRLHF, an open-source framework enabling efficient
  RLHF scaling. Unlike existing RLHF frameworks that co-locate four models on the
  same GPUs, OpenRLHF re-designs scheduling for the models beyond 70B parameters
  using Ray, vLLM, and DeepSpeed, leveraging improved resource utilization and
  diverse training approaches. Integrating seamlessly with Hugging Face, OpenRLHF
  provides an out-of-the-box solution with optimized algorithms and launch
  scripts, which ensures user-friendliness. OpenRLHF implements RLHF, DPO,
  rejection sampling, and other alignment techniques. Empowering state-of-the-art
  LLM development, OpenRLHF's code is available at
  https://github.com/OpenLLMAI/OpenRLHF.
paper_ai_summary: OpenRLHF is an open-source framework that optimizes the scaling of reinforcement learning from human feedback (RLHF) for large language models by improving resource utilization and integrating with Hugging Face.
paper_ai_keywords: LLMs reinforcement learning from human feedback RLHF Ray vLLM DeepSpeed Hugging Face DPO rejection sampling