Secrets of RLHF in Large Language Models Part I: PPO
Large language models (LLMs) have formulated a blueprint for the advancement
of artificial general intelligence. Its primary objective is to function as a
human-centric (helpful, honest, and harmless) assistant. Alignment with humans
assumes paramount significance, and reinforcement learning with human feedback
(RLHF) emerges as the pivotal technological paradigm underpinning this pursuit.
Current technical routes usually include reward models to measure
human preferences, Proximal Policy Optimization (PPO) to optimize
policy model outputs, and process supervision to improve step-by-step
reasoning capabilities. However, due to the challenges of reward design,
environment interaction, and agent training, coupled with huge trial and error
cost of large language models, there is a significant barrier for AI
researchers to motivate the development of technical alignment and safe landing
of LLMs. The stable training of RLHF has still been a puzzle. In the first
report, we dissect the framework of RLHF, re-evaluate the inner workings of
PPO, and explore how the parts comprising PPO algorithms impact policy agent
training. We identify policy constraints being the key factor for the effective
implementation of the PPO algorithm. Therefore, we explore the PPO-max, an
advanced version of PPO algorithm, to efficiently improve the training
stability of the policy model. Based on our main results, we perform a
comprehensive analysis of RLHF abilities compared with SFT models and ChatGPT.
The absence of open-source implementations has posed significant challenges to
the investigation of LLMs alignment. Therefore, we are eager to release
technical reports, reward models and PPO codes
64adfa72616d3eb3616367ed
Rui Zheng
64adfa72616d3eb3616367ee
Shihan Dou
admin_assigned
2023-07-13T08:34:48.046Z
64ae0b6ccf90fe27556cf56e
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/uVXJd3TBssT_Ug4yAl3c9.jpeg
Shihan Dou
user
Ablustrund
64adfa72616d3eb3616367ef
Songyang Gao
64adfa72616d3eb3616367f0
Wei Shen
claimed_verified
2024-06-28T08:03:41.077Z
62308ae8be7833fa98620372
/avatars/220ed57376202c61ec1afbf5a4f45b67.svg
random
user
fakerbaby
64adfa72616d3eb3616367f1
Binghai Wang
admin_assigned
2023-07-13T08:35:55.941Z
642127bbe40f66bcd1e7e047
/avatars/8da3b0ae9723af8d96906c7307e90bb8.svg
wang binghai
user
refrain-wbh
64adfa72616d3eb3616367f2
Yan Liu
64adfa72616d3eb3616367f3
Senjie Jin
64adfa72616d3eb3616367f4
Qin Liu
64adfa72616d3eb3616367f5
Limao Xiong
64adfa72616d3eb3616367f6
Lu Chen
64adfa72616d3eb3616367f7
Zhiheng Xi
claimed_verified
2024-10-25T09:33:00.421Z
653a6e5cae155b92bae77b74
https://cdn-avatars.huggingface.co/v1/production/uploads/653a6e5cae155b92bae77b74/TA5FWKAUsB249ux4MzD_R.jpeg
Zhiheng Xi
user
WooooDyy
64adfa72616d3eb3616367f8
Yuhao Zhou
64adfa72616d3eb3616367f9
Nuo Xu
64adfa72616d3eb3616367fa
Wenbin Lai
64adfa72616d3eb3616367fb
Minghao Zhu
64adfa72616d3eb3616367fc
Rongxiang Weng
64adfa72616d3eb3616367fd
Wensen Cheng
64adfa72616d3eb3616367fe
Cheng Chang
64adfa72616d3eb3616367ff
Zhangyue Yin
admin_assigned
2023-07-13T10:05:15.094Z
628c5da32f09ccf530204dbe
https://cdn-avatars.huggingface.co/v1/production/uploads/1653366416287-628c5da32f09ccf530204dbe.jpeg
Zhangyue Yin
user
yinzhangyue
64adfa72616d3eb361636800
Yuan Hua
64adfa72616d3eb361636801
Haoran Huang
64adfa72616d3eb361636802
Tianxiang Sun
admin_assigned
2023-07-13T10:06:01.528Z
618c9e4cf35289d4bcab9d02
/avatars/28b5b096190b539295ec0da167f1abf3.svg
Tianxiang Sun
user
txsun
64adfa72616d3eb361636803
Hang Yan
64adfa72616d3eb361636804
Tao Gui
64adfa72616d3eb361636805
Qi Zhang
64adfa72616d3eb361636806
Xipeng Qiu
admin_assigned
2023-07-13T10:07:56.582Z
61457b8deff2c9fdb4de4988
https://cdn-avatars.huggingface.co/v1/production/uploads/1632381702899-61457b8deff2c9fdb4de4988.jpeg
Xipeng Qiu
user
xpqiu
64adfa72616d3eb361636807
Xuanjing Huang
Secrets of RLHF in Large Language Models Part I: PPO
Large language models (LLMs) have formulated a blueprint for the advancement
of artificial general intelligence. Its primary objective is to function as a
human-centric (helpful, honest, and harmless) assistant. Alignment with humans
assumes paramount significance, and reinforcement learning with human feedback
(RLHF) emerges as the pivotal technological paradigm underpinning this pursuit.
Current technical routes usually include reward models to measure
human preferences, Proximal Policy Optimization (PPO) to optimize
policy model outputs, and process supervision to improve step-by-step
reasoning capabilities. However, due to the challenges of reward design,
environment interaction, and agent training, coupled with huge trial and error
cost of large language models, there is a significant barrier for AI
researchers to motivate the development of technical alignment and safe landing
of LLMs. The stable training of RLHF has still been a puzzle. In the first
report, we dissect the framework of RLHF, re-evaluate the inner workings of
PPO, and explore how the parts comprising PPO algorithms impact policy agent
training. We identify policy constraints being the key factor for the effective
implementation of the PPO algorithm. Therefore, we explore the PPO-max, an
advanced version of PPO algorithm, to efficiently improve the training
stability of the policy model. Based on our main results, we perform a
comprehensive analysis of RLHF abilities compared with SFT models and ChatGPT.
The absence of open-source implementations has posed significant challenges to
the investigation of LLMs alignment. Therefore, we are eager to release
technical reports, reward models and PPO codes
This report examines Reinfocement Learning with Human Feedback (RLHF) and proposes PPO-max to improve the stability of policy model training compared to other SFT models and ChatGPT.
Reinfocement Learning with Human Feedback (RLHF) reward models Proximal Policy Optimization (PPO) policy constraints PPO-max step-by-step reasoning policy model training Stable Training Safe Landing