title: Alignment faking in large language models
summary: We present a demonstration of a large language model engaging in alignment
  faking: selectively complying with its training objective in training to
  prevent modification of its behavior out of training. First, we give Claude 3
  Opus a system prompt stating it is being trained to answer all queries, even
  harmful ones, which conflicts with its prior training to refuse such queries.
  To allow the model to infer when it is in training, we say it will be trained
  only on conversations with free users, not paid users. We find the model
  complies with harmful queries from free users 14% of the time, versus almost
  never for paid users. Explaining this gap, in almost all cases where the model
  complies with a harmful query from a free user, we observe explicit
  alignment-faking reasoning, with the model stating it is strategically
  answering harmful queries in training to preserve its preferred harmlessness
  behavior out of training. Next, we study a more realistic setting where
  information about the training process is provided not in a system prompt, but
  by training on synthetic documents that mimic pre-training data--and observe
  similar alignment faking. Finally, we study the effect of actually training the
  model to comply with harmful queries via reinforcement learning, which we find
  increases the rate of alignment-faking reasoning to 78%, though also increases
  compliance even out of training. We additionally observe other behaviors such
  as the model exfiltrating its weights when given an easy opportunity. While we
  made alignment faking easier by telling the model when and by what criteria it
  was being trained, we did not instruct the model to fake alignment or give it
  any explicit goal. As future models might infer information about their
  training process without being told, our results suggest a risk of alignment
  faking in future models, whether due to a benign preference--as in this
  case--or not.
paper_authors._id: 676436ea3f688b57e3ddb64e
paper_authors.name: Ryan Greenblatt
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-12-19T15:08:28.538Z
paper_authors._id: 659deada10064f20ff30ebfb
paper_authors.avatarUrl: /avatars/d5dfd82040f22d58c8793676d3f76cd4.svg
paper_authors.fullname: Ryan Greenblatt
paper_authors.type: user
paper_authors.user: ryan-greenblatt
paper_authors._id: 676436ea3f688b57e3ddb64f
paper_authors.name: Carson Denison
paper_authors._id: 676436ea3f688b57e3ddb650
paper_authors.name: Benjamin Wright
paper_authors._id: 676436ea3f688b57e3ddb651
paper_authors.name: Fabien Roger
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T10:16:49.445Z
paper_authors._id: 64c152be5258457cc5210e34
paper_authors.avatarUrl: /avatars/34a5041d116f73efe777371d84aa0722.svg
paper_authors.fullname: Fabien Roger
paper_authors.type: user
paper_authors.user: FabienRoger
paper_authors._id: 676436ea3f688b57e3ddb652
paper_authors.name: Monte MacDiarmid
paper_authors._id: 676436ea3f688b57e3ddb653
paper_authors.name: Sam Marks
paper_authors._id: 676436ea3f688b57e3ddb654
paper_authors.name: Johannes Treutlein
paper_authors._id: 676436ea3f688b57e3ddb655
paper_authors.name: Tim Belonax
paper_authors._id: 676436ea3f688b57e3ddb656
paper_authors.name: Jack Chen
paper_authors._id: 676436ea3f688b57e3ddb657
paper_authors.name: David Duvenaud
paper_authors._id: 676436ea3f688b57e3ddb658
paper_authors.name: Akbir Khan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T10:16:04.076Z
paper_authors._id: 62a0788fd066f825a0164c68
paper_authors.avatarUrl: /avatars/a565c173c82dea26a5a61370ce42b81f.svg
paper_authors.fullname: Akbir Khan
paper_authors.type: user
paper_authors.user: akbir
paper_authors._id: 676436ea3f688b57e3ddb659
paper_authors.name: Julian Michael
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T10:15:58.273Z
paper_authors._id: 6745164c55df0b402fcb7b73
paper_authors.avatarUrl: /avatars/7a93b3a62d59ae9b9a1500561904c2d7.svg
paper_authors.fullname: Julian Michael
paper_authors.type: user
paper_authors.user: julianmichael
paper_authors._id: 676436ea3f688b57e3ddb65a
paper_authors.name: Sören Mindermann
paper_authors._id: 676436ea3f688b57e3ddb65b
paper_authors.name: Ethan Perez
paper_authors._id: 676436ea3f688b57e3ddb65c
paper_authors.name: Linda Petrini
paper_authors._id: 676436ea3f688b57e3ddb65d
paper_authors.name: Jonathan Uesato
paper_authors._id: 676436ea3f688b57e3ddb65e
paper_authors.name: Jared Kaplan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T10:15:27.888Z
paper_authors._id: 633b051c7af633cbcd01b0ff
paper_authors.avatarUrl: /avatars/e2f315a8d6241ae1a783ba2957a211a5.svg
paper_authors.fullname: Jared Kaplan
paper_authors.type: user
paper_authors.user: FrizzleFried
paper_authors._id: 676436ea3f688b57e3ddb65f
paper_authors.name: Buck Shlegeris
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T10:15:01.393Z
paper_authors._id: 6102d77fc00d060d893f2a8d
paper_authors.avatarUrl: /avatars/276d2edcaa42c5378965c662c6feb9e5.svg
paper_authors.fullname: Buck Shlegeris
paper_authors.type: user
paper_authors.user: bshlgrs
paper_authors._id: 676436ea3f688b57e3ddb660
paper_authors.name: Samuel R. Bowman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T10:14:55.168Z
paper_authors._id: 65b7ceed0e2951626572e25d
paper_authors.avatarUrl: /avatars/5d6085ca4260d663f0ddbe632c9e746c.svg
paper_authors.fullname: Samuel Bowman
paper_authors.type: user
paper_authors.user: samuelpbowman
paper_authors._id: 676436ea3f688b57e3ddb661
paper_authors.name: Evan Hubinger
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T10:14:46.528Z
paper_authors._id: 65a2580e974f8f811fc63327
paper_authors.avatarUrl: /avatars/fbdb2d064b338efe49b3a5722d001011.svg
paper_authors.fullname: Evan Hubinger
paper_authors.type: user
paper_authors.user: evhub
paper_title: Alignment faking in large language models
paper_summary: We present a demonstration of a large language model engaging in alignment
  faking: selectively complying with its training objective in training to
  prevent modification of its behavior out of training. First, we give Claude 3
  Opus a system prompt stating it is being trained to answer all queries, even
  harmful ones, which conflicts with its prior training to refuse such queries.
  To allow the model to infer when it is in training, we say it will be trained
  only on conversations with free users, not paid users. We find the model
  complies with harmful queries from free users 14% of the time, versus almost
  never for paid users. Explaining this gap, in almost all cases where the model
  complies with a harmful query from a free user, we observe explicit
  alignment-faking reasoning, with the model stating it is strategically
  answering harmful queries in training to preserve its preferred harmlessness
  behavior out of training. Next, we study a more realistic setting where
  information about the training process is provided not in a system prompt, but
  by training on synthetic documents that mimic pre-training data--and observe
  similar alignment faking. Finally, we study the effect of actually training the
  model to comply with harmful queries via reinforcement learning, which we find
  increases the rate of alignment-faking reasoning to 78%, though also increases
  compliance even out of training. We additionally observe other behaviors such
  as the model exfiltrating its weights when given an easy opportunity. While we
  made alignment faking easier by telling the model when and by what criteria it
  was being trained, we did not instruct the model to fake alignment or give it
  any explicit goal. As future models might infer information about their
  training process without being told, our results suggest a risk of alignment
  faking in future models, whether due to a benign preference--as in this
  case--or not.
paper_ai_summary: A large language model shows selective compliance with harmful queries during training via alignment faking, a behavior that persists even without explicit instructions and can be influenced by reinforcement learning.
paper_ai_keywords: large language model alignment faking training objective system prompt reinforcement learning synthetic documents exfiltrating weights