title: PERL: Parameter Efficient Reinforcement Learning from Human Feedback
summary: Reinforcement Learning from Human Feedback (RLHF) has proven to be a strong
  method to align Pretrained Large Language Models (LLMs) with human preferences.
  But training models with RLHF is computationally expensive, and an overall
  complex process. In this work, we study RLHF where the underlying models are
  trained using the parameter efficient method of Low-Rank Adaptation (LoRA)
  introduced by Hu et al. [2021]. We investigate the setup of "Parameter
  Efficient Reinforcement Learning" (PERL), in which we perform reward model
  training and reinforcement learning using LoRA. We compare PERL to conventional
  fine-tuning (full-tuning) across various configurations for 7 benchmarks,
  including 2 novel datasets, of reward modeling and reinforcement learning. We
  find that PERL performs on par with the conventional RLHF setting, while
  training faster, and with less memory. This enables the high performance of
  RLHF, while reducing the computational burden that limits its adoption as an
  alignment technique for Large Language Models. We also release 2 novel thumbs
  up/down preference datasets: "Taskmaster Coffee", and "Taskmaster Ticketing" to
  promote research around RLHF.
paper_authors._id: 65f91d16ea2d72e3b584f191
paper_authors.name: Hakim Sidahmed
paper_authors._id: 65f91d16ea2d72e3b584f192
paper_authors.name: Samrat Phatale
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-19T09:30:47.424Z
paper_authors._id: 64fb65f348411fc78979a654
paper_authors.avatarUrl: /avatars/66b61203f2a6fd07f9026ba97f9e8c5d.svg
paper_authors.fullname: Samrat Phatale
paper_authors.type: user
paper_authors.user: samratph
paper_authors._id: 65f91d16ea2d72e3b584f193
paper_authors.name: Alex Hutcheson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-19T09:30:53.475Z
paper_authors._id: 65e8e8213ef9a491d42d7f36
paper_authors.avatarUrl: /avatars/35e63bfd293ee4a31f4057122c403ed0.svg
paper_authors.fullname: alexhutcheson
paper_authors.type: user
paper_authors.user: alexhutcheson
paper_authors._id: 65f91d16ea2d72e3b584f194
paper_authors.name: Zhuonan Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-19T20:44:30.657Z
paper_authors._id: 65f9ef2b3260327fc7033467
paper_authors.avatarUrl: /avatars/623ef7c11543b5009cb2d6b8ad2e082f.svg
paper_authors.fullname: Zhuonan Lin
paper_authors.type: user
paper_authors.user: zhuonanlin
paper_authors._id: 65f91d16ea2d72e3b584f195
paper_authors.name: Zhang Chen
paper_authors._id: 65f91d16ea2d72e3b584f196
paper_authors.name: Zac Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-19T09:31:43.363Z
paper_authors._id: 6436815a0b2c0d86d0b183fc
paper_authors.avatarUrl: /avatars/fc9d7825e5f108f0ec231c9f4d1caeb6.svg
paper_authors.fullname: zac yu
paper_authors.type: user
paper_authors.user: zacyy
paper_authors._id: 65f91d16ea2d72e3b584f197
paper_authors.name: Jarvis Jin
paper_authors._id: 65f91d16ea2d72e3b584f198
paper_authors.name: Roman Komarytsia
paper_authors._id: 65f91d16ea2d72e3b584f199
paper_authors.name: Christiane Ahlheim
paper_authors._id: 65f91d16ea2d72e3b584f19a
paper_authors.name: Yonghao Zhu
paper_authors._id: 65f91d16ea2d72e3b584f19b
paper_authors.name: Simral Chaudhary
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-21T08:27:53.457Z
paper_authors._id: 65fa130087d1c912d9944f4e
paper_authors.avatarUrl: /avatars/8976decb5c7706e8007d707d655cd82a.svg
paper_authors.fullname: Simral Chaudhary
paper_authors.type: user
paper_authors.user: simral
paper_authors._id: 65f91d16ea2d72e3b584f19c
paper_authors.name: Bowen Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-25T09:44:20.411Z
paper_authors._id: 65a8b8689db09e1fc1502b7f
paper_authors.avatarUrl: /avatars/bba2faab9c040a7aba40c53773b37e21.svg
paper_authors.fullname: Bowen Li
paper_authors.type: user
paper_authors.user: bowenlee919
paper_authors._id: 65f91d16ea2d72e3b584f19d
paper_authors.name: Saravanan Ganesh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-19T09:32:51.302Z
paper_authors._id: 6450077817b6189cda58fbcd
paper_authors.avatarUrl: /avatars/df47f880b3d7c824b1a616c39ebdfb1c.svg
paper_authors.fullname: Saravanan Ganesh
paper_authors.type: user
paper_authors.user: srrvnn
paper_authors._id: 65f91d16ea2d72e3b584f19e
paper_authors.name: Bill Byrne
paper_authors._id: 65f91d16ea2d72e3b584f19f
paper_authors.name: Jessica Hoffmann
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-19T09:33:03.861Z
paper_authors._id: 6429734a8136224fee06a749
paper_authors.avatarUrl: /avatars/dafe1e85f3d97ab9acd5547fd2919ba9.svg
paper_authors.fullname: Jessica Hoffmann
paper_authors.type: user
paper_authors.user: jessicahoffmann111
paper_authors._id: 65f91d16ea2d72e3b584f1a0
paper_authors.name: Hassan Mansoor
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-19T09:33:09.298Z
paper_authors._id: 629098fd5463575364e7697a
paper_authors.avatarUrl: /avatars/e45f3ee28bff3571f76caf847d3c36db.svg
paper_authors.fullname: Hassan Mansoor
paper_authors.type: user
paper_authors.user: HassanMansoor
paper_authors._id: 65f91d16ea2d72e3b584f1a1
paper_authors.name: Wei Li
paper_authors._id: 65f91d16ea2d72e3b584f1a2
paper_authors.name: Abhinav Rastogi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-26T07:55:13.485Z
paper_authors._id: 653966932a101a151da1155c
paper_authors.avatarUrl: /avatars/10baa6536793b397c05028011d4ffcdc.svg
paper_authors.fullname: Abhinav Rastogi
paper_authors.type: user
paper_authors.user: abhirast
paper_authors._id: 65f91d16ea2d72e3b584f1a3
paper_authors.name: Lucas Dixon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-19T09:33:50.021Z
paper_authors._id: 64c25a9c2b1ffc2eecbce31d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c25a9c2b1ffc2eecbce31d/uTp3yOPdmCSAmhzWwKUjK.png
paper_authors.fullname: Lucas Dixon
paper_authors.type: user
paper_authors.user: iislucas
paper_title: PERL: Parameter Efficient Reinforcement Learning from Human Feedback
paper_summary: Reinforcement Learning from Human Feedback (RLHF) has proven to be a strong
  method to align Pretrained Large Language Models (LLMs) with human preferences.
  But training models with RLHF is computationally expensive, and an overall
  complex process. In this work, we study RLHF where the underlying models are
  trained using the parameter efficient method of Low-Rank Adaptation (LoRA)
  introduced by Hu et al. [2021]. We investigate the setup of "Parameter
  Efficient Reinforcement Learning" (PERL), in which we perform reward model
  training and reinforcement learning using LoRA. We compare PERL to conventional
  fine-tuning (full-tuning) across various configurations for 7 benchmarks,
  including 2 novel datasets, of reward modeling and reinforcement learning. We
  find that PERL performs on par with the conventional RLHF setting, while
  training faster, and with less memory. This enables the high performance of
  RLHF, while reducing the computational burden that limits its adoption as an
  alignment technique for Large Language Models. We also release 2 novel thumbs
  up/down preference datasets: "Taskmaster Coffee", and "Taskmaster Ticketing" to
  promote research around RLHF.
paper_ai_keywords: Reinforcement Learning from Human Feedback (RLHF) Pretrained Large Language Models (LLMs) Low-Rank Adaptation (LoRA) Parameter Efficient Reinforcement Learning (PERL) reward model training reinforcement learning full-tuning Taskmaster Coffee Taskmaster Ticketing