title: RLHF Workflow: From Reward Modeling to Online RLHF
summary: We present the workflow of Online Iterative Reinforcement Learning from Human
  Feedback (RLHF) in this technical report, which is widely reported to
  outperform its offline counterpart by a large margin in the recent large
  language model (LLM) literature. However, existing open-source RLHF projects
  are still largely confined to the offline learning setting. In this technical
  report, we aim to fill in this gap and provide a detailed recipe that is easy
  to reproduce for online iterative RLHF. In particular, since online human
  feedback is usually infeasible for open-source communities with limited
  resources, we start by constructing preference models using a diverse set of
  open-source datasets and use the constructed proxy preference model to
  approximate human feedback. Then, we discuss the theoretical insights and
  algorithmic principles behind online iterative RLHF, followed by a detailed
  practical implementation. Our trained LLM, SFR-Iterative-DPO-LLaMA-3-8B-R,
  achieves impressive performance on LLM chatbot benchmarks, including
  AlpacaEval-2, Arena-Hard, and MT-Bench, as well as other academic benchmarks
  such as HumanEval and TruthfulQA. We have shown that supervised fine-tuning
  (SFT) and iterative RLHF can obtain state-of-the-art performance with fully
  open-source datasets. Further, we have made our models, curated datasets, and
  comprehensive step-by-step code guidebooks publicly available. Please refer to
  https://github.com/RLHFlow/RLHF-Reward-Modeling and
  https://github.com/RLHFlow/Online-RLHF for more detailed information.
paper_authors._id: 6642e6abe4a7476619c240f1
paper_authors.name: Hanze Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-14T07:12:07.320Z
paper_authors._id: 63a3ff69f91ad3ea5703841d
paper_authors.avatarUrl: /avatars/69227c4bce01d33747c1377b6f9672db.svg
paper_authors.fullname: Hanze Dong
paper_authors.type: user
paper_authors.user: hendrydong
paper_authors._id: 6642e6abe4a7476619c240f2
paper_authors.name: Wei Xiong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-14T07:53:00.588Z
paper_authors._id: 643e59806db6ba8c5ee123f3
paper_authors.avatarUrl: /avatars/4052f2a250107f43b3634c3ee3cc30a1.svg
paper_authors.fullname: Wei Xiong
paper_authors.type: user
paper_authors.user: weqweasdas
paper_authors._id: 6642e6abe4a7476619c240f3
paper_authors.name: Bo Pang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-14T07:12:38.677Z
paper_authors._id: 63e08acbf351dc0745749d56
paper_authors.avatarUrl: /avatars/8e2d5ce9db5bd8008ac2ad80f6025553.svg
paper_authors.fullname: Bo Pang
paper_authors.type: user
paper_authors.user: bpucla
paper_authors._id: 6642e6abe4a7476619c240f4
paper_authors.name: Haoxiang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-14T07:13:11.114Z
paper_authors._id: 638fb8cf2380ffd99caf8c2a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638fb8cf2380ffd99caf8c2a/7juUomHJ4gH1HIJj43qE6.jpeg
paper_authors.fullname: Haoxiang Wang
paper_authors.type: user
paper_authors.user: Haoxiang-Wang
paper_authors._id: 6642e6abe4a7476619c240f5
paper_authors.name: Han Zhao
paper_authors._id: 6642e6abe4a7476619c240f6
paper_authors.name: Yingbo Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-14T07:13:32.770Z
paper_authors._id: 649bc93758d8b19de0c7785f
paper_authors.avatarUrl: /avatars/3ed9473aee23d99f4ee949d3705089ea.svg
paper_authors.fullname: Yingbo Zhou
paper_authors.type: user
paper_authors.user: yingbozhou
paper_authors._id: 6642e6abe4a7476619c240f7
paper_authors.name: Nan Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-14T07:14:02.032Z
paper_authors._id: 64b8922ca1827cc8d04ae919
paper_authors.avatarUrl: /avatars/0aaa83e3d09a82434e1d6af724aaa485.svg
paper_authors.fullname: Nan Jiang
paper_authors.type: user
paper_authors.user: nanjiang
paper_authors._id: 6642e6abe4a7476619c240f8
paper_authors.name: Doyen Sahoo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-14T07:14:08.581Z
paper_authors._id: 65f84fd980481173afd91233
paper_authors.avatarUrl: /avatars/6ac7bd6beba24d1476c5179b88c9e3fa.svg
paper_authors.fullname: Doyen
paper_authors.type: user
paper_authors.user: doyensahoo
paper_authors._id: 6642e6abe4a7476619c240f9
paper_authors.name: Caiming Xiong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-14T07:14:15.835Z
paper_authors._id: 649dbcc4e0fff1ed099dc80a
paper_authors.avatarUrl: /avatars/c87c273ca628dbcddccbf1ee19b2ce33.svg
paper_authors.fullname: Caiming Xiong
paper_authors.type: user
paper_authors.user: cxiong
paper_authors._id: 6642e6abe4a7476619c240fa
paper_authors.name: Tong Zhang
paper_title: RLHF Workflow: From Reward Modeling to Online RLHF
paper_summary: We present the workflow of Online Iterative Reinforcement Learning from Human
  Feedback (RLHF) in this technical report, which is widely reported to
  outperform its offline counterpart by a large margin in the recent large
  language model (LLM) literature. However, existing open-source RLHF projects
  are still largely confined to the offline learning setting. In this technical
  report, we aim to fill in this gap and provide a detailed recipe that is easy
  to reproduce for online iterative RLHF. In particular, since online human
  feedback is usually infeasible for open-source communities with limited
  resources, we start by constructing preference models using a diverse set of
  open-source datasets and use the constructed proxy preference model to
  approximate human feedback. Then, we discuss the theoretical insights and
  algorithmic principles behind online iterative RLHF, followed by a detailed
  practical implementation. Our trained LLM, SFR-Iterative-DPO-LLaMA-3-8B-R,
  achieves impressive performance on LLM chatbot benchmarks, including
  AlpacaEval-2, Arena-Hard, and MT-Bench, as well as other academic benchmarks
  such as HumanEval and TruthfulQA. We have shown that supervised fine-tuning
  (SFT) and iterative RLHF can obtain state-of-the-art performance with fully
  open-source datasets. Further, we have made our models, curated datasets, and
  comprehensive step-by-step code guidebooks publicly available. Please refer to
  https://github.com/RLHFlow/RLHF-Reward-Modeling and
  https://github.com/RLHFlow/Online-RLHF for more detailed information.
paper_ai_keywords: Reinforcement Learning from Human Feedback (RLHF) Online Iterative Reinforcement Learning (RLHF) preference models proxy preference model online learning supervised fine-tuning (SFT) iterative RLHF LLM chatbot benchmarks AlpacaEval-2 Arena-Hard MT-Bench HumanEval TruthfulQA