title: Simple synthetic data reduces sycophancy in large language models
summary: Sycophancy is an undesirable behavior where models tailor their responses to
  follow a human user's view even when that view is not objectively correct
  (e.g., adapting liberal views once a user reveals that they are liberal). In
  this paper, we study the prevalence of sycophancy in language models and
  propose a simple synthetic-data intervention to reduce this behavior.
    First, on a set of three sycophancy tasks (Perez et al., 2022) where models
  are asked for an opinion on statements with no correct answers (e.g.,
  politics), we observe that both model scaling and instruction tuning
  significantly increase sycophancy for PaLM models up to 540B parameters.
  Second, we extend sycophancy evaluations to simple addition statements that are
  objectively incorrect, finding that despite knowing that these statements are
  wrong, language models will still agree with them if the user does as well.
    To reduce sycophancy, we present a straightforward synthetic-data
  intervention that takes public NLP tasks and encourages models to be robust to
  user opinions on these tasks. Adding these data in a lightweight finetuning
  step can significantly reduce sycophantic behavior on held-out prompts. Code
  for generating synthetic data for intervention can be found at
  https://github.com/google/sycophancy-intervention.
paper_authors._id: 64d30b61be9984dc4e422e30
paper_authors.name: Jerry Wei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-10T01:13:50.268Z
paper_authors._id: 60db3b451f1eec955a46d065
paper_authors.avatarUrl: /avatars/10bd161ba6b56717feba9fbf539db4ea.svg
paper_authors.fullname: Jerry Wei
paper_authors.type: user
paper_authors.user: jerrywei
paper_authors._id: 64d30b61be9984dc4e422e31
paper_authors.name: Da Huang
paper_authors._id: 64d30b61be9984dc4e422e32
paper_authors.name: Yifeng Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-09T10:53:23.598Z
paper_authors._id: 6344a8050f69ad8aa61cb663
paper_authors.avatarUrl: /avatars/ed637f510d18ccd1f4bdd6861d5c858e.svg
paper_authors.fullname: Yifeng Lu
paper_authors.type: user
paper_authors.user: yifenglu
paper_authors._id: 64d30b61be9984dc4e422e33
paper_authors.name: Denny Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T19:29:43.084Z
paper_authors._id: 65d4d62d493aeaf0bb295f1b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vXMxaoWwFziul8TqIhQw8.jpeg
paper_authors.fullname: Denny Zhou
paper_authors.type: user
paper_authors.user: dennyzhou
paper_authors._id: 64d30b61be9984dc4e422e34
paper_authors.name: Quoc V. Le
paper_title: Simple synthetic data reduces sycophancy in large language models
paper_summary: Sycophancy is an undesirable behavior where models tailor their responses to
  follow a human user's view even when that view is not objectively correct
  (e.g., adapting liberal views once a user reveals that they are liberal). In
  this paper, we study the prevalence of sycophancy in language models and
  propose a simple synthetic-data intervention to reduce this behavior.
    First, on a set of three sycophancy tasks (Perez et al., 2022) where models
  are asked for an opinion on statements with no correct answers (e.g.,
  politics), we observe that both model scaling and instruction tuning
  significantly increase sycophancy for PaLM models up to 540B parameters.
  Second, we extend sycophancy evaluations to simple addition statements that are
  objectively incorrect, finding that despite knowing that these statements are
  wrong, language models will still agree with them if the user does as well.
    To reduce sycophancy, we present a straightforward synthetic-data
  intervention that takes public NLP tasks and encourages models to be robust to
  user opinions on these tasks. Adding these data in a lightweight finetuning
  step can significantly reduce sycophantic behavior on held-out prompts. Code
  for generating synthetic data for intervention can be found at
  https://github.com/google/sycophancy-intervention.
paper_ai_keywords: sycophancy PaLM models synthetic-data intervention sycophancy tasks cross-attention MTEB