OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework
As large language models (LLMs) continue to grow by scaling laws,
reinforcement learning from human feedback (RLHF) has gained significant
attention due to its outstanding performance. However, unlike pretraining or
fine-tuning a single model, scaling reinforcement learning from human feedback
(RLHF) for training large language models poses coordination challenges across
four models. We present OpenRLHF, an open-source framework enabling efficient
RLHF scaling. Unlike existing RLHF frameworks that co-locate four models on the
same GPUs, OpenRLHF re-designs scheduling for the models beyond 70B parameters
using Ray, vLLM, and DeepSpeed, leveraging improved resource utilization and
diverse training approaches. Integrating seamlessly with Hugging Face, OpenRLHF
provides an out-of-the-box solution with optimized algorithms and launch
scripts, which ensures user-friendliness. OpenRLHF implements RLHF, DPO,
rejection sampling, and other alignment techniques. Empowering state-of-the-art
LLM development, OpenRLHF's code is available at
https://github.com/OpenLLMAI/OpenRLHF.
664c0b558e6575f45e13d2ad
Jian Hu
extracted_confirmed
2025-06-05T00:20:51.693Z
63f6c04ac96958470d1e9043
/avatars/da46cdd9e21498e120ca91b67bfbfb5e.svg
Jian Hu
user
chuyi777
664c0b558e6575f45e13d2ae
Xibin Wu
admin_assigned
2024-05-21T08:13:06.288Z
64f5a6b46e335217131d7d99
/avatars/c4a15eadc5e612947a314a4a66a51c75.svg
Xibin Wu
user
wuxibin
664c0b558e6575f45e13d2af
Weixun Wang
claimed_verified
2024-05-21T08:59:18.059Z
6263541963f73be3d2f91bdd
/avatars/57474711c7e7768169509eb70e47176e.svg
weixun
user
weixun
664c0b558e6575f45e13d2b0
Xianyu
claimed_verified
2024-05-21T14:15:45.678Z
6054383110aba34e3b6a2f02
/avatars/63ff3a2ce469da0f59e9ef501250a136.svg
Xianyu
user
catqaq
664c0b558e6575f45e13d2b1
Dehao Zhang
admin_assigned
2024-05-21T08:14:28.286Z
634d0eeffc4873ed0d00406a
https://cdn-avatars.huggingface.co/v1/production/uploads/1665994473058-noauth.jpeg
ZhangDehao
user
ispoon
664c0b558e6575f45e13d2b2
Yu Cao
claimed_verified
2024-05-27T07:20:12.327Z
64618ad756c57120a2d1990c
/avatars/1339e57e4d314116a72191d2d2d88f2c.svg
Tony Cao
user
Tony991
OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework
As large language models (LLMs) continue to grow by scaling laws,
reinforcement learning from human feedback (RLHF) has gained significant
attention due to its outstanding performance. However, unlike pretraining or
fine-tuning a single model, scaling reinforcement learning from human feedback
(RLHF) for training large language models poses coordination challenges across
four models. We present OpenRLHF, an open-source framework enabling efficient
RLHF scaling. Unlike existing RLHF frameworks that co-locate four models on the
same GPUs, OpenRLHF re-designs scheduling for the models beyond 70B parameters
using Ray, vLLM, and DeepSpeed, leveraging improved resource utilization and
diverse training approaches. Integrating seamlessly with Hugging Face, OpenRLHF
provides an out-of-the-box solution with optimized algorithms and launch
scripts, which ensures user-friendliness. OpenRLHF implements RLHF, DPO,
rejection sampling, and other alignment techniques. Empowering state-of-the-art
LLM development, OpenRLHF's code is available at
https://github.com/OpenLLMAI/OpenRLHF.
OpenRLHF is an open-source framework that optimizes the scaling of reinforcement learning from human feedback (RLHF) for large language models by improving resource utilization and integrating with Hugging Face.
LLMs reinforcement learning from human feedback RLHF Ray vLLM DeepSpeed Hugging Face DPO rejection sampling