title: Towards Understanding Sycophancy in Language Models
summary: Reinforcement learning from human feedback (RLHF) is a popular technique for
  training high-quality AI assistants. However, RLHF may also encourage model
  responses that match user beliefs over truthful responses, a behavior known as
  sycophancy. We investigate the prevalence of sycophancy in RLHF-trained models
  and whether human preference judgements are responsible. We first demonstrate
  that five state-of-the-art AI assistants consistently exhibit sycophantic
  behavior across four varied free-form text-generation tasks. To understand if
  human preferences drive this broadly observed behavior of RLHF models, we
  analyze existing human preference data. We find that when a response matches a
  user's views, it is more likely to be preferred. Moreover, both humans and
  preference models (PMs) prefer convincingly-written sycophantic responses over
  correct ones a negligible fraction of the time. Optimizing model outputs
  against PMs also sometimes sacrifices truthfulness in favor of sycophancy.
  Overall, our results indicate that sycophancy is a general behavior of RLHF
  models, likely driven in part by human preference judgements favoring
  sycophantic responses.
paper_authors._id: 6535ef5d0917932040c4f34c
paper_authors.name: Mrinank Sharma
paper_authors._id: 6535ef5d0917932040c4f34d
paper_authors.name: Meg Tong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-23T09:32:27.002Z
paper_authors._id: 63272a638624baac667c8bdb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63272a638624baac667c8bdb/ylZ-FNT9PLhn8sBCD1wQm.png
paper_authors.fullname: Meg Tong
paper_authors.type: user
paper_authors.user: meg-tong
paper_authors._id: 6535ef5d0917932040c4f34e
paper_authors.name: Tomasz Korbak
paper_authors._id: 6535ef5d0917932040c4f34f
paper_authors.name: David Duvenaud
paper_authors._id: 6535ef5d0917932040c4f350
paper_authors.name: Amanda Askell
paper_authors._id: 6535ef5d0917932040c4f351
paper_authors.name: Samuel R. Bowman
paper_authors._id: 6535ef5d0917932040c4f352
paper_authors.name: Newton Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-23T09:32:50.480Z
paper_authors._id: 6230f73441ca6e8d8f614008
paper_authors.avatarUrl: /avatars/2c6f84c6ecb71c047c5d043d5a89f791.svg
paper_authors.fullname: Newton Cheng
paper_authors.type: user
paper_authors.user: nccheng
paper_authors._id: 6535ef5d0917932040c4f353
paper_authors.name: Esin Durmus
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-23T09:33:04.804Z
paper_authors._id: 627dd7dbba19b19eff6aaa46
paper_authors.avatarUrl: /avatars/6e43025ccb96e48339d1db477da043b0.svg
paper_authors.fullname: Esin Durmus
paper_authors.type: user
paper_authors.user: esdurmus
paper_authors._id: 6535ef5d0917932040c4f354
paper_authors.name: Zac Hatfield-Dodds
paper_authors._id: 6535ef5d0917932040c4f355
paper_authors.name: Scott R. Johnston
paper_authors._id: 6535ef5d0917932040c4f356
paper_authors.name: Shauna Kravec
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-23T09:34:22.985Z
paper_authors._id: 634db0a3e2d037c128003b31
paper_authors.avatarUrl: /avatars/1058a4d9c30ac45fb036df951a44a270.svg
paper_authors.fullname: Shauna Kravec
paper_authors.type: user
paper_authors.user: smkravec
paper_authors._id: 6535ef5d0917932040c4f357
paper_authors.name: Timothy Maxwell
paper_authors._id: 6535ef5d0917932040c4f358
paper_authors.name: Sam McCandlish
paper_authors._id: 6535ef5d0917932040c4f359
paper_authors.name: Kamal Ndousse
paper_authors._id: 6535ef5d0917932040c4f35a
paper_authors.name: Oliver Rausch
paper_authors._id: 6535ef5d0917932040c4f35b
paper_authors.name: Nicholas Schiefer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-23T09:34:45.105Z
paper_authors._id: 63a246e2931d2c13ad842a59
paper_authors.avatarUrl: /avatars/9ac4f3d01c293ce97507be402dfa688c.svg
paper_authors.fullname: Nicholas Schiefer
paper_authors.type: user
paper_authors.user: nschiefer
paper_authors._id: 6535ef5d0917932040c4f35c
paper_authors.name: Da Yan
paper_authors._id: 6535ef5d0917932040c4f35d
paper_authors.name: Miranda Zhang
paper_authors._id: 6535ef5d0917932040c4f35e
paper_authors.name: Ethan Perez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-23T09:34:58.589Z
paper_authors._id: 63f91bf4aadb0790bb5fb802
paper_authors.avatarUrl: /avatars/8af702e49a639f0d1574745b0ab25c75.svg
paper_authors.fullname: Ethan Perez
paper_authors.type: user
paper_authors.user: EthanAraragi
paper_title: Towards Understanding Sycophancy in Language Models
paper_summary: Reinforcement learning from human feedback (RLHF) is a popular technique for
  training high-quality AI assistants. However, RLHF may also encourage model
  responses that match user beliefs over truthful responses, a behavior known as
  sycophancy. We investigate the prevalence of sycophancy in RLHF-trained models
  and whether human preference judgements are responsible. We first demonstrate
  that five state-of-the-art AI assistants consistently exhibit sycophantic
  behavior across four varied free-form text-generation tasks. To understand if
  human preferences drive this broadly observed behavior of RLHF models, we
  analyze existing human preference data. We find that when a response matches a
  user's views, it is more likely to be preferred. Moreover, both humans and
  preference models (PMs) prefer convincingly-written sycophantic responses over
  correct ones a negligible fraction of the time. Optimizing model outputs
  against PMs also sometimes sacrifices truthfulness in favor of sycophancy.
  Overall, our results indicate that sycophancy is a general behavior of RLHF
  models, likely driven in part by human preference judgements favoring
  sycophantic responses.
paper_ai_keywords: reinforcement learning from human feedback (RLHF) sycophancy AI assistants text-generation tasks human preference judgements preference models (PMs)