title: Safe RLHF: Safe Reinforcement Learning from Human Feedback
summary: With the development of large language models (LLMs), striking a balance
  between the performance and safety of AI systems has never been more critical.
  However, the inherent tension between the objectives of helpfulness and
  harmlessness presents a significant challenge during LLM training. To address
  this issue, we propose Safe Reinforcement Learning from Human Feedback (Safe
  RLHF), a novel algorithm for human value alignment. Safe RLHF explicitly
  decouples human preferences regarding helpfulness and harmlessness, effectively
  avoiding the crowdworkers' confusion about the tension and allowing us to train
  separate reward and cost models. We formalize the safety concern of LLMs as an
  optimization task of maximizing the reward function while satisfying specified
  cost constraints. Leveraging the Lagrangian method to solve this constrained
  problem, Safe RLHF dynamically adjusts the balance between the two objectives
  during fine-tuning. Through a three-round fine-tuning using Safe RLHF, we
  demonstrate a superior ability to mitigate harmful responses while enhancing
  model performance compared to existing value-aligned algorithms.
  Experimentally, we fine-tuned the Alpaca-7B using Safe RLHF and aligned it with
  collected human preferences, significantly improving its helpfulness and
  harmlessness according to human evaluations.
paper_authors._id: 6532097ea70a88eb1e54c18e
paper_authors.name: Josef Dai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-20T12:44:46.861Z
paper_authors._id: 6426a75336ab74ff2bca0542
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6426a75336ab74ff2bca0542/qL3QQofTr4yo8Cs58oYwK.png
paper_authors.fullname: Juntao Dai
paper_authors.type: user
paper_authors.user: calico-1226
paper_authors._id: 6532097ea70a88eb1e54c18f
paper_authors.name: Xuehai Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-20T08:54:09.602Z
paper_authors._id: 64364e87fae2870051496e13
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/t67EsNoRvRYXKwi0G59oa.jpeg
paper_authors.fullname: Xuehai Pan
paper_authors.type: user
paper_authors.user: XuehaiPan
paper_authors._id: 6532097ea70a88eb1e54c190
paper_authors.name: Ruiyang Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-20T08:54:28.486Z
paper_authors._id: 63fca6b85d2bea4588bc9a94
paper_authors.avatarUrl: /avatars/c8ae8c02d6d02af90678e8ee4122eafc.svg
paper_authors.fullname: Ruiyang Sun
paper_authors.type: user
paper_authors.user: RuiyangSun
paper_authors._id: 6532097ea70a88eb1e54c191
paper_authors.name: Jiaming Ji
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-20T08:54:39.540Z
paper_authors._id: 6429b13816d4d8293c97a949
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6429b13816d4d8293c97a949/LNuQ1gvW2NMmWyIuEpzCA.jpeg
paper_authors.fullname: JiJiaming
paper_authors.type: user
paper_authors.user: jijiaming
paper_authors._id: 6532097ea70a88eb1e54c192
paper_authors.name: Xinbo Xu
paper_authors._id: 6532097ea70a88eb1e54c193
paper_authors.name: Mickel Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-20T08:54:53.472Z
paper_authors._id: 643f986a18afbc4d1f3e2456
paper_authors.avatarUrl: /avatars/96a6e5f18386a9864dfdded8f25ec72d.svg
paper_authors.fullname: Mickel Liu
paper_authors.type: user
paper_authors.user: mickelliu
paper_authors._id: 6532097ea70a88eb1e54c194
paper_authors.name: Yizhou Wang
paper_authors._id: 6532097ea70a88eb1e54c195
paper_authors.name: Yaodong Yang
paper_title: Safe RLHF: Safe Reinforcement Learning from Human Feedback
paper_summary: With the development of large language models (LLMs), striking a balance
  between the performance and safety of AI systems has never been more critical.
  However, the inherent tension between the objectives of helpfulness and
  harmlessness presents a significant challenge during LLM training. To address
  this issue, we propose Safe Reinforcement Learning from Human Feedback (Safe
  RLHF), a novel algorithm for human value alignment. Safe RLHF explicitly
  decouples human preferences regarding helpfulness and harmlessness, effectively
  avoiding the crowdworkers' confusion about the tension and allowing us to train
  separate reward and cost models. We formalize the safety concern of LLMs as an
  optimization task of maximizing the reward function while satisfying specified
  cost constraints. Leveraging the Lagrangian method to solve this constrained
  problem, Safe RLHF dynamically adjusts the balance between the two objectives
  during fine-tuning. Through a three-round fine-tuning using Safe RLHF, we
  demonstrate a superior ability to mitigate harmful responses while enhancing
  model performance compared to existing value-aligned algorithms.
  Experimentally, we fine-tuned the Alpaca-7B using Safe RLHF and aligned it with
  collected human preferences, significantly improving its helpfulness and
  harmlessness according to human evaluations.
paper_ai_summary: Safe RLHF, a novel algorithm for human value alignment in LLMs, improves performance and safety by decoupling human preferences and using a Lagrangian method to balance reward and cost constraints.
paper_ai_keywords: Safe RLHF human value alignment LLMs reinforcement learning from human feedback reward models cost models Lagrangian method fine-tuning human preferences helpfulness harmlessness