Efficient RLHF: Reducing the Memory Usage of PPO
Reinforcement Learning with Human Feedback (RLHF) has revolutionized language
modeling by aligning models with human preferences. However, the RL stage,
Proximal Policy Optimization (PPO), requires over 3x the memory of Supervised
Fine-Tuning (SFT), making it infeasible to use for most practitioners. To
address this issue, we present a comprehensive analysis the memory usage,
performance, and training time of memory-savings techniques for PPO. We
introduce Hydra-RLHF by first integrating the SFT and Reward models and then
dynamically turning LoRA "off" during training. Our experiments show: 1. Using
LoRA during PPO reduces its memory usage to be smaller than SFT while improving
alignment across four public benchmarks, and 2. Hydra-PPO reduces the latency
per sample of LoRA-PPO by up to 65% while maintaining its performance. Our
results demonstrate that Hydra-PPO is a simple and promising solution for
enabling more widespread usage of RLHF.
64f7ebe4d493d8b0d2a0606b
Michael Santacroce
admin_assigned
2023-09-06T11:22:58.527Z
63e3de93737f1e423752cec6
/avatars/e7340ed2696535c8aa0e3291fa2cfdc8.svg
Michael Santacroce
user
santacml
64f7ebe4d493d8b0d2a0606c
Yadong Lu
64f7ebe4d493d8b0d2a0606d
Han Yu
64f7ebe4d493d8b0d2a0606e
Yuanzhi Li
admin_assigned
2023-09-06T11:24:43.228Z
641d5a313f04f9bf2da17e8c
/avatars/4035f73c69ddb045573ad96db4f13f04.svg
Yuanzhi Li
user
Uushizhu1234
64f7ebe4d493d8b0d2a0606f
Yelong Shen
admin_assigned
2023-09-06T11:25:07.791Z
6454c337a13edf669cd5d8ea
/avatars/a383a0dda7c2ef6a0d6c3c64651f42ff.svg
Yelong Shen
user
uuu6
Efficient RLHF: Reducing the Memory Usage of PPO
Reinforcement Learning with Human Feedback (RLHF) has revolutionized language
modeling by aligning models with human preferences. However, the RL stage,
Proximal Policy Optimization (PPO), requires over 3x the memory of Supervised
Fine-Tuning (SFT), making it infeasible to use for most practitioners. To
address this issue, we present a comprehensive analysis the memory usage,
performance, and training time of memory-savings techniques for PPO. We
introduce Hydra-RLHF by first integrating the SFT and Reward models and then
dynamically turning LoRA "off" during training. Our experiments show: 1. Using
LoRA during PPO reduces its memory usage to be smaller than SFT while improving
alignment across four public benchmarks, and 2. Hydra-PPO reduces the latency
per sample of LoRA-PPO by up to 65% while maintaining its performance. Our
results demonstrate that Hydra-PPO is a simple and promising solution for
enabling more widespread usage of RLHF.
Hydra-RLHF optimizes Reinforcement Learning with Human Feedback by integrating SFT and Reward models and dynamically disabling LoRA, reducing memory usage and latency while maintaining performance.
Reinforcement Learning with Human Feedback RLHF Proximal Policy Optimization PPO Supervised Fine-Tuning SFT Reward models LoRA Hydra-RLHF Hydra-PPO