title: Secrets of RLHF in Large Language Models Part I: PPO
summary: Large language models (LLMs) have formulated a blueprint for the advancement
  of artificial general intelligence. Its primary objective is to function as a
  human-centric (helpful, honest, and harmless) assistant. Alignment with humans
  assumes paramount significance, and reinforcement learning with human feedback
  (RLHF) emerges as the pivotal technological paradigm underpinning this pursuit.
  Current technical routes usually include reward models to measure
  human preferences, Proximal Policy Optimization (PPO) to optimize
  policy model outputs, and process supervision to improve step-by-step
  reasoning capabilities. However, due to the challenges of reward design,
  environment interaction, and agent training, coupled with huge trial and error
  cost of large language models, there is a significant barrier for AI
  researchers to motivate the development of technical alignment and safe landing
  of LLMs. The stable training of RLHF has still been a puzzle. In the first
  report, we dissect the framework of RLHF, re-evaluate the inner workings of
  PPO, and explore how the parts comprising PPO algorithms impact policy agent
  training. We identify policy constraints being the key factor for the effective
  implementation of the PPO algorithm. Therefore, we explore the PPO-max, an
  advanced version of PPO algorithm, to efficiently improve the training
  stability of the policy model. Based on our main results, we perform a
  comprehensive analysis of RLHF abilities compared with SFT models and ChatGPT.
  The absence of open-source implementations has posed significant challenges to
  the investigation of LLMs alignment. Therefore, we are eager to release
  technical reports, reward models and PPO codes
paper_authors._id: 64adfa72616d3eb3616367ed
paper_authors.name: Rui Zheng
paper_authors._id: 64adfa72616d3eb3616367ee
paper_authors.name: Shihan Dou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-13T08:34:48.046Z
paper_authors._id: 64ae0b6ccf90fe27556cf56e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/uVXJd3TBssT_Ug4yAl3c9.jpeg
paper_authors.fullname: Shihan Dou
paper_authors.type: user
paper_authors.user: Ablustrund
paper_authors._id: 64adfa72616d3eb3616367ef
paper_authors.name: Songyang Gao
paper_authors._id: 64adfa72616d3eb3616367f0
paper_authors.name: Wei Shen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-28T08:03:41.077Z
paper_authors._id: 62308ae8be7833fa98620372
paper_authors.avatarUrl: /avatars/220ed57376202c61ec1afbf5a4f45b67.svg
paper_authors.fullname: random
paper_authors.type: user
paper_authors.user: fakerbaby
paper_authors._id: 64adfa72616d3eb3616367f1
paper_authors.name: Binghai Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-13T08:35:55.941Z
paper_authors._id: 642127bbe40f66bcd1e7e047
paper_authors.avatarUrl: /avatars/8da3b0ae9723af8d96906c7307e90bb8.svg
paper_authors.fullname: wang binghai
paper_authors.type: user
paper_authors.user: refrain-wbh
paper_authors._id: 64adfa72616d3eb3616367f2
paper_authors.name: Yan Liu
paper_authors._id: 64adfa72616d3eb3616367f3
paper_authors.name: Senjie Jin
paper_authors._id: 64adfa72616d3eb3616367f4
paper_authors.name: Qin Liu
paper_authors._id: 64adfa72616d3eb3616367f5
paper_authors.name: Limao Xiong
paper_authors._id: 64adfa72616d3eb3616367f6
paper_authors.name: Lu Chen
paper_authors._id: 64adfa72616d3eb3616367f7
paper_authors.name: Zhiheng Xi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-25T09:33:00.421Z
paper_authors._id: 653a6e5cae155b92bae77b74
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/653a6e5cae155b92bae77b74/TA5FWKAUsB249ux4MzD_R.jpeg
paper_authors.fullname: Zhiheng Xi
paper_authors.type: user
paper_authors.user: WooooDyy
paper_authors._id: 64adfa72616d3eb3616367f8
paper_authors.name: Yuhao Zhou
paper_authors._id: 64adfa72616d3eb3616367f9
paper_authors.name: Nuo Xu
paper_authors._id: 64adfa72616d3eb3616367fa
paper_authors.name: Wenbin Lai
paper_authors._id: 64adfa72616d3eb3616367fb
paper_authors.name: Minghao Zhu
paper_authors._id: 64adfa72616d3eb3616367fc
paper_authors.name: Rongxiang Weng
paper_authors._id: 64adfa72616d3eb3616367fd
paper_authors.name: Wensen Cheng
paper_authors._id: 64adfa72616d3eb3616367fe
paper_authors.name: Cheng Chang
paper_authors._id: 64adfa72616d3eb3616367ff
paper_authors.name: Zhangyue Yin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-13T10:05:15.094Z
paper_authors._id: 628c5da32f09ccf530204dbe
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1653366416287-628c5da32f09ccf530204dbe.jpeg
paper_authors.fullname: Zhangyue Yin
paper_authors.type: user
paper_authors.user: yinzhangyue
paper_authors._id: 64adfa72616d3eb361636800
paper_authors.name: Yuan Hua
paper_authors._id: 64adfa72616d3eb361636801
paper_authors.name: Haoran Huang
paper_authors._id: 64adfa72616d3eb361636802
paper_authors.name: Tianxiang Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-13T10:06:01.528Z
paper_authors._id: 618c9e4cf35289d4bcab9d02
paper_authors.avatarUrl: /avatars/28b5b096190b539295ec0da167f1abf3.svg
paper_authors.fullname: Tianxiang Sun
paper_authors.type: user
paper_authors.user: txsun
paper_authors._id: 64adfa72616d3eb361636803
paper_authors.name: Hang Yan
paper_authors._id: 64adfa72616d3eb361636804
paper_authors.name: Tao Gui
paper_authors._id: 64adfa72616d3eb361636805
paper_authors.name: Qi Zhang
paper_authors._id: 64adfa72616d3eb361636806
paper_authors.name: Xipeng Qiu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-13T10:07:56.582Z
paper_authors._id: 61457b8deff2c9fdb4de4988
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1632381702899-61457b8deff2c9fdb4de4988.jpeg
paper_authors.fullname: Xipeng Qiu
paper_authors.type: user
paper_authors.user: xpqiu
paper_authors._id: 64adfa72616d3eb361636807
paper_authors.name: Xuanjing Huang
paper_title: Secrets of RLHF in Large Language Models Part I: PPO
paper_summary: Large language models (LLMs) have formulated a blueprint for the advancement
  of artificial general intelligence. Its primary objective is to function as a
  human-centric (helpful, honest, and harmless) assistant. Alignment with humans
  assumes paramount significance, and reinforcement learning with human feedback
  (RLHF) emerges as the pivotal technological paradigm underpinning this pursuit.
  Current technical routes usually include reward models to measure
  human preferences, Proximal Policy Optimization (PPO) to optimize
  policy model outputs, and process supervision to improve step-by-step
  reasoning capabilities. However, due to the challenges of reward design,
  environment interaction, and agent training, coupled with huge trial and error
  cost of large language models, there is a significant barrier for AI
  researchers to motivate the development of technical alignment and safe landing
  of LLMs. The stable training of RLHF has still been a puzzle. In the first
  report, we dissect the framework of RLHF, re-evaluate the inner workings of
  PPO, and explore how the parts comprising PPO algorithms impact policy agent
  training. We identify policy constraints being the key factor for the effective
  implementation of the PPO algorithm. Therefore, we explore the PPO-max, an
  advanced version of PPO algorithm, to efficiently improve the training
  stability of the policy model. Based on our main results, we perform a
  comprehensive analysis of RLHF abilities compared with SFT models and ChatGPT.
  The absence of open-source implementations has posed significant challenges to
  the investigation of LLMs alignment. Therefore, we are eager to release
  technical reports, reward models and PPO codes
paper_ai_summary: This report examines Reinfocement Learning with Human Feedback (RLHF) and proposes PPO-max to improve the stability of policy model training compared to other SFT models and ChatGPT.
paper_ai_keywords: Reinfocement Learning with Human Feedback (RLHF) reward models Proximal Policy Optimization (PPO) policy constraints PPO-max step-by-step reasoning policy model training Stable Training Safe Landing