title: Open Problems and Fundamental Limitations of Reinforcement Learning from
    Human Feedback
summary: Reinforcement learning from human feedback (RLHF) is a technique for training
  AI systems to align with human goals. RLHF has emerged as the central method
  used to finetune state-of-the-art large language models (LLMs). Despite this
  popularity, there has been relatively little public work systematizing its
  flaws. In this paper, we (1) survey open problems and fundamental limitations
  of RLHF and related methods; (2) overview techniques to understand, improve,
  and complement RLHF in practice; and (3) propose auditing and disclosure
  standards to improve societal oversight of RLHF systems. Our work emphasizes
  the limitations of RLHF and highlights the importance of a multi-faceted
  approach to the development of safer AI systems.
paper_authors._id: 64c74566589826641bcfccd7
paper_authors.name: Stephen Casper
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-31T06:21:48.917Z
paper_authors._id: 6466a046326128fd2c6c59c2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6466a046326128fd2c6c59c2/kYwPcxupelOBvKFB0y8Me.png
paper_authors.fullname: Stephen Casper
paper_authors.type: user
paper_authors.user: stecas
paper_authors._id: 64c74566589826641bcfccd8
paper_authors.name: Xander Davies
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-31T06:21:57.219Z
paper_authors._id: 62d689078faee0ac953b97c4
paper_authors.avatarUrl: /avatars/1c35b888bb36046439ab00a18c609eb1.svg
paper_authors.fullname: Alexander Davies
paper_authors.type: user
paper_authors.user: xanderdavies
paper_authors._id: 64c74566589826641bcfccd9
paper_authors.name: Claudia Shi
paper_authors._id: 64c74566589826641bcfccda
paper_authors.name: Thomas Krendl Gilbert
paper_authors._id: 64c74566589826641bcfccdb
paper_authors.name: Jérémy Scheurer
paper_authors._id: 64c74566589826641bcfccdc
paper_authors.name: Javier Rando
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-31T06:23:27.907Z
paper_authors._id: 6245b1b2d5895bdf34ef8c33
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6245b1b2d5895bdf34ef8c33/hLmcppWpRS7s2rxMxdWGh.jpeg
paper_authors.fullname: Javier Rando
paper_authors.type: user
paper_authors.user: javirandor
paper_authors._id: 64c74566589826641bcfccdd
paper_authors.name: Rachel Freedman
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-01T07:44:55.055Z
paper_authors._id: 64c7d8173d5a0dfed5a612ce
paper_authors.avatarUrl: /avatars/738b6113e35420e3d20f277db8c38416.svg
paper_authors.fullname: Rachel Freedman
paper_authors.type: user
paper_authors.user: RachelFreedman
paper_authors._id: 64c74566589826641bcfccde
paper_authors.name: Tomasz Korbak
paper_authors._id: 64c74566589826641bcfccdf
paper_authors.name: David Lindner
paper_authors._id: 64c74566589826641bcfcce0
paper_authors.name: Pedro Freire
paper_authors._id: 64c74566589826641bcfcce1
paper_authors.name: Tony Wang
paper_authors._id: 64c74566589826641bcfcce2
paper_authors.name: Samuel Marks
paper_authors._id: 64c74566589826641bcfcce3
paper_authors.name: Charbel-Raphaël Segerie
paper_authors._id: 64c74566589826641bcfcce4
paper_authors.name: Micah Carroll
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-01T07:44:51.608Z
paper_authors._id: 625cba98aa13d02f2a8d9029
paper_authors.avatarUrl: /avatars/3b557357ee9ae07dd12ba930f3942e17.svg
paper_authors.fullname: Carroll
paper_authors.type: user
paper_authors.user: Micah
paper_authors._id: 64c74566589826641bcfcce5
paper_authors.name: Andi Peng
paper_authors._id: 64c74566589826641bcfcce6
paper_authors.name: Phillip Christoffersen
paper_authors._id: 64c74566589826641bcfcce7
paper_authors.name: Mehul Damani
paper_authors._id: 64c74566589826641bcfcce8
paper_authors.name: Stewart Slocum
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-31T06:26:55.278Z
paper_authors._id: 64b71864efefc8a73877c2c0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b71864efefc8a73877c2c0/Xdwn0KsSNev-iBrlnXh0o.jpeg
paper_authors.fullname: Stewart Slocum
paper_authors.type: user
paper_authors.user: stewy33
paper_authors._id: 64c74566589826641bcfcce9
paper_authors.name: Usman Anwar
paper_authors._id: 64c74566589826641bcfccea
paper_authors.name: Anand Siththaranjan
paper_authors._id: 64c74566589826641bcfcceb
paper_authors.name: Max Nadeau
paper_authors._id: 64c74566589826641bcfccec
paper_authors.name: Eric J. Michaud
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-24T12:03:37.262Z
paper_authors._id: 6303e59c7b50dd9d0a367375
paper_authors.avatarUrl: /avatars/353aaaae6db1c7dc89895927a65fa9b1.svg
paper_authors.fullname: Eric Michaud
paper_authors.type: user
paper_authors.user: ericjm
paper_authors._id: 64c74566589826641bcfcced
paper_authors.name: Jacob Pfau
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-31T06:47:31.412Z
paper_authors._id: 63766f562361581ceb23d34f
paper_authors.avatarUrl: /avatars/8563015693f5e0e03c25d39034cae599.svg
paper_authors.fullname: Jacob Pfau
paper_authors.type: user
paper_authors.user: jacobpfau
paper_authors._id: 64c74566589826641bcfccee
paper_authors.name: Dmitrii Krasheninnikov
paper_authors._id: 64c74566589826641bcfccef
paper_authors.name: Xin Chen
paper_authors._id: 64c74566589826641bcfccf0
paper_authors.name: Lauro Langosco
paper_authors._id: 64c74566589826641bcfccf1
paper_authors.name: Peter Hase
paper_authors._id: 64c74566589826641bcfccf2
paper_authors.name: Erdem Bıyık
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-31T06:49:18.212Z
paper_authors._id: 644387fe9504687e2d82f0e5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/wx1emudpMZ0BzdnqI7Xnl.png
paper_authors.fullname: Erdem Bıyık
paper_authors.type: user
paper_authors.user: ebiyik
paper_authors._id: 64c74566589826641bcfccf3
paper_authors.name: Anca Dragan
paper_authors._id: 64c74566589826641bcfccf4
paper_authors.name: David Krueger
paper_authors._id: 64c74566589826641bcfccf5
paper_authors.name: Dorsa Sadigh
paper_authors._id: 64c74566589826641bcfccf6
paper_authors.name: Dylan Hadfield-Menell
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-31T06:50:02.742Z
paper_authors._id: 64905f22dc0d89dca8f0db13
paper_authors.avatarUrl: /avatars/7e30391d6b8914806724afbcd7401a29.svg
paper_authors.fullname: Dylan Hadfield-Menell
paper_authors.type: user
paper_authors.user: dhadfieldmenell
paper_title: Open Problems and Fundamental Limitations of Reinforcement Learning from
    Human Feedback
paper_summary: Reinforcement learning from human feedback (RLHF) is a technique for training
  AI systems to align with human goals. RLHF has emerged as the central method
  used to finetune state-of-the-art large language models (LLMs). Despite this
  popularity, there has been relatively little public work systematizing its
  flaws. In this paper, we (1) survey open problems and fundamental limitations
  of RLHF and related methods; (2) overview techniques to understand, improve,
  and complement RLHF in practice; and (3) propose auditing and disclosure
  standards to improve societal oversight of RLHF systems. Our work emphasizes
  the limitations of RLHF and highlights the importance of a multi-faceted
  approach to the development of safer AI systems.