title: Efficient RLHF: Reducing the Memory Usage of PPO
summary: Reinforcement Learning with Human Feedback (RLHF) has revolutionized language
  modeling by aligning models with human preferences. However, the RL stage,
  Proximal Policy Optimization (PPO), requires over 3x the memory of Supervised
  Fine-Tuning (SFT), making it infeasible to use for most practitioners. To
  address this issue, we present a comprehensive analysis the memory usage,
  performance, and training time of memory-savings techniques for PPO. We
  introduce Hydra-RLHF by first integrating the SFT and Reward models and then
  dynamically turning LoRA "off" during training. Our experiments show: 1. Using
  LoRA during PPO reduces its memory usage to be smaller than SFT while improving
  alignment across four public benchmarks, and 2. Hydra-PPO reduces the latency
  per sample of LoRA-PPO by up to 65% while maintaining its performance. Our
  results demonstrate that Hydra-PPO is a simple and promising solution for
  enabling more widespread usage of RLHF.
paper_authors._id: 64f7ebe4d493d8b0d2a0606b
paper_authors.name: Michael Santacroce
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-06T11:22:58.527Z
paper_authors._id: 63e3de93737f1e423752cec6
paper_authors.avatarUrl: /avatars/e7340ed2696535c8aa0e3291fa2cfdc8.svg
paper_authors.fullname: Michael Santacroce
paper_authors.type: user
paper_authors.user: santacml
paper_authors._id: 64f7ebe4d493d8b0d2a0606c
paper_authors.name: Yadong Lu
paper_authors._id: 64f7ebe4d493d8b0d2a0606d
paper_authors.name: Han Yu
paper_authors._id: 64f7ebe4d493d8b0d2a0606e
paper_authors.name: Yuanzhi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-06T11:24:43.228Z
paper_authors._id: 641d5a313f04f9bf2da17e8c
paper_authors.avatarUrl: /avatars/4035f73c69ddb045573ad96db4f13f04.svg
paper_authors.fullname: Yuanzhi Li
paper_authors.type: user
paper_authors.user: Uushizhu1234
paper_authors._id: 64f7ebe4d493d8b0d2a0606f
paper_authors.name: Yelong Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-06T11:25:07.791Z
paper_authors._id: 6454c337a13edf669cd5d8ea
paper_authors.avatarUrl: /avatars/a383a0dda7c2ef6a0d6c3c64651f42ff.svg
paper_authors.fullname: Yelong Shen
paper_authors.type: user
paper_authors.user: uuu6
paper_title: Efficient RLHF: Reducing the Memory Usage of PPO
paper_summary: Reinforcement Learning with Human Feedback (RLHF) has revolutionized language
  modeling by aligning models with human preferences. However, the RL stage,
  Proximal Policy Optimization (PPO), requires over 3x the memory of Supervised
  Fine-Tuning (SFT), making it infeasible to use for most practitioners. To
  address this issue, we present a comprehensive analysis the memory usage,
  performance, and training time of memory-savings techniques for PPO. We
  introduce Hydra-RLHF by first integrating the SFT and Reward models and then
  dynamically turning LoRA "off" during training. Our experiments show: 1. Using
  LoRA during PPO reduces its memory usage to be smaller than SFT while improving
  alignment across four public benchmarks, and 2. Hydra-PPO reduces the latency
  per sample of LoRA-PPO by up to 65% while maintaining its performance. Our
  results demonstrate that Hydra-PPO is a simple and promising solution for
  enabling more widespread usage of RLHF.
paper_ai_summary: Hydra-RLHF optimizes Reinforcement Learning with Human Feedback by integrating SFT and Reward models and dynamically disabling LoRA, reducing memory usage and latency while maintaining performance.
paper_ai_keywords: Reinforcement Learning with Human Feedback RLHF Proximal Policy Optimization PPO Supervised Fine-Tuning SFT Reward models LoRA Hydra-RLHF Hydra-PPO