title: Sleeper Agents: Training Deceptive LLMs that Persist Through Safety
    Training
summary: Humans are capable of strategically deceptive behavior: behaving helpfully in
  most situations, but then behaving very differently in order to pursue
  alternative objectives when given the opportunity. If an AI system learned such
  a deceptive strategy, could we detect it and remove it using current
  state-of-the-art safety training techniques? To study this question, we
  construct proof-of-concept examples of deceptive behavior in large language
  models (LLMs). For example, we train models that write secure code when the
  prompt states that the year is 2023, but insert exploitable code when the
  stated year is 2024. We find that such backdoored behavior can be made
  persistent, so that it is not removed by standard safety training techniques,
  including supervised fine-tuning, reinforcement learning, and adversarial
  training (eliciting unsafe behavior and then training to remove it). The
  backdoored behavior is most persistent in the largest models and in models
  trained to produce chain-of-thought reasoning about deceiving the training
  process, with the persistence remaining even when the chain-of-thought is
  distilled away. Furthermore, rather than removing backdoors, we find that
  adversarial training can teach models to better recognize their backdoor
  triggers, effectively hiding the unsafe behavior. Our results suggest that,
  once a model exhibits deceptive behavior, standard techniques could fail to
  remove such deception and create a false impression of safety.
paper_authors._id: 65a0a37f9e18386f471f3f7d
paper_authors.name: Evan Hubinger
paper_authors._id: 65a0a37f9e18386f471f3f7e
paper_authors.name: Carson Denison
paper_authors._id: 65a0a37f9e18386f471f3f7f
paper_authors.name: Jesse Mu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:32:12.408Z
paper_authors._id: 62301010174feb5439c42e23
paper_authors.avatarUrl: /avatars/cd3c8a97823e3cbc176fef245113624f.svg
paper_authors.fullname: Jesse Mu
paper_authors.type: user
paper_authors.user: jayelm
paper_authors._id: 65a0a37f9e18386f471f3f80
paper_authors.name: Mike Lambert
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:32:36.618Z
paper_authors._id: 62996d51b58e71e2aca23964
paper_authors.avatarUrl: /avatars/1d71b80340b11a46e70cab86590b459e.svg
paper_authors.fullname: Mike Lambert
paper_authors.type: user
paper_authors.user: MikeLambert
paper_authors._id: 65a0a37f9e18386f471f3f81
paper_authors.name: Meg Tong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:32:43.288Z
paper_authors._id: 63272a638624baac667c8bdb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63272a638624baac667c8bdb/ylZ-FNT9PLhn8sBCD1wQm.png
paper_authors.fullname: Meg Tong
paper_authors.type: user
paper_authors.user: meg-tong
paper_authors._id: 65a0a37f9e18386f471f3f82
paper_authors.name: Monte MacDiarmid
paper_authors._id: 65a0a37f9e18386f471f3f83
paper_authors.name: Tamera Lanham
paper_authors._id: 65a0a37f9e18386f471f3f84
paper_authors.name: Daniel M. Ziegler
paper_authors._id: 65a0a37f9e18386f471f3f85
paper_authors.name: Tim Maxwell
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:34:04.934Z
paper_authors._id: 6573ad29addd9129ff33819b
paper_authors.avatarUrl: /avatars/792d426a6c306c41fa0818e63e7aeefb.svg
paper_authors.fullname: Tim Maxwell
paper_authors.type: user
paper_authors.user: maxwellinked
paper_authors._id: 65a0a37f9e18386f471f3f86
paper_authors.name: Newton Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:34:21.109Z
paper_authors._id: 6230f73441ca6e8d8f614008
paper_authors.avatarUrl: /avatars/2c6f84c6ecb71c047c5d043d5a89f791.svg
paper_authors.fullname: Newton Cheng
paper_authors.type: user
paper_authors.user: nccheng
paper_authors._id: 65a0a37f9e18386f471f3f87
paper_authors.name: Adam Jermyn
paper_authors._id: 65a0a37f9e18386f471f3f88
paper_authors.name: Amanda Askell
paper_authors._id: 65a0a37f9e18386f471f3f89
paper_authors.name: Ansh Radhakrishnan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:34:37.230Z
paper_authors._id: 625e0426e1176e6c772aa37c
paper_authors.avatarUrl: /avatars/a05c5e3b964ca2a6b82eec0e3d89c064.svg
paper_authors.fullname: Ansh Radhakrishnan
paper_authors.type: user
paper_authors.user: anshr
paper_authors._id: 65a0a37f9e18386f471f3f8a
paper_authors.name: Cem Anil
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:34:44.023Z
paper_authors._id: 62f6b27d05ca68c0e0008549
paper_authors.avatarUrl: /avatars/14d0075aa1b578cd7ee5f9e68d12e2f0.svg
paper_authors.fullname: Cem Anil
paper_authors.type: user
paper_authors.user: anilcem
paper_authors._id: 65a0a37f9e18386f471f3f8b
paper_authors.name: David Duvenaud
paper_authors._id: 65a0a37f9e18386f471f3f8c
paper_authors.name: Deep Ganguli
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:34:55.651Z
paper_authors._id: 63924e6b2bfa48413aa0a4ad
paper_authors.avatarUrl: /avatars/f8219c475af6995b6fb9ebe727c5be43.svg
paper_authors.fullname: Deep Ganguli
paper_authors.type: user
paper_authors.user: dganguli
paper_authors._id: 65a0a37f9e18386f471f3f8d
paper_authors.name: Fazl Barez
paper_authors._id: 65a0a37f9e18386f471f3f8e
paper_authors.name: Jack Clark
paper_authors._id: 65a0a37f9e18386f471f3f8f
paper_authors.name: Kamal Ndousse
paper_authors._id: 65a0a37f9e18386f471f3f90
paper_authors.name: Kshitij Sachan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:35:43.559Z
paper_authors._id: 64d51ddea146b1c0a642ca95
paper_authors.avatarUrl: /avatars/6448e205712ad2375fbba79a32cb62a1.svg
paper_authors.fullname: Kshitij Sachan
paper_authors.type: user
paper_authors.user: ksachan
paper_authors._id: 65a0a37f9e18386f471f3f91
paper_authors.name: Michael Sellitto
paper_authors._id: 65a0a37f9e18386f471f3f92
paper_authors.name: Mrinank Sharma
paper_authors._id: 65a0a37f9e18386f471f3f93
paper_authors.name: Nova DasSarma
paper_authors._id: 65a0a37f9e18386f471f3f94
paper_authors.name: Roger Grosse
paper_authors._id: 65a0a37f9e18386f471f3f95
paper_authors.name: Shauna Kravec
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:36:11.235Z
paper_authors._id: 634db0a3e2d037c128003b31
paper_authors.avatarUrl: /avatars/1058a4d9c30ac45fb036df951a44a270.svg
paper_authors.fullname: Shauna Kravec
paper_authors.type: user
paper_authors.user: smkravec
paper_authors._id: 65a0a37f9e18386f471f3f96
paper_authors.name: Yuntao Bai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:36:24.369Z
paper_authors._id: 636ba29fac42fe6b1bcb3686
paper_authors.avatarUrl: /avatars/6ec793667e225616ffbb69ae7d57566e.svg
paper_authors.fullname: bai
paper_authors.type: user
paper_authors.user: baiyuntao
paper_authors._id: 65a0a37f9e18386f471f3f97
paper_authors.name: Zachary Witten
paper_authors._id: 65a0a37f9e18386f471f3f98
paper_authors.name: Marina Favaro
paper_authors._id: 65a0a37f9e18386f471f3f99
paper_authors.name: Jan Brauner
paper_authors._id: 65a0a37f9e18386f471f3f9a
paper_authors.name: Holden Karnofsky
paper_authors._id: 65a0a37f9e18386f471f3f9b
paper_authors.name: Paul Christiano
paper_authors._id: 65a0a37f9e18386f471f3f9c
paper_authors.name: Samuel R. Bowman
paper_authors._id: 65a0a37f9e18386f471f3f9d
paper_authors.name: Logan Graham
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:37:01.310Z
paper_authors._id: 6379a6332cb50dda39d99926
paper_authors.avatarUrl: /avatars/385427fe5fd7043abcc814ef50db6f4a.svg
paper_authors.fullname: Logan Graham
paper_authors.type: user
paper_authors.user: logangraham
paper_authors._id: 65a0a37f9e18386f471f3f9e
paper_authors.name: Jared Kaplan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:37:09.365Z
paper_authors._id: 633b051c7af633cbcd01b0ff
paper_authors.avatarUrl: /avatars/e2f315a8d6241ae1a783ba2957a211a5.svg
paper_authors.fullname: Jared Kaplan
paper_authors.type: user
paper_authors.user: FrizzleFried
paper_authors._id: 65a0a37f9e18386f471f3f9f
paper_authors.name: Sören Mindermann
paper_authors._id: 65a0a37f9e18386f471f3fa0
paper_authors.name: Ryan Greenblatt
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:37:23.118Z
paper_authors._id: 659deada10064f20ff30ebfb
paper_authors.avatarUrl: /avatars/d5dfd82040f22d58c8793676d3f76cd4.svg
paper_authors.fullname: Ryan Greenblatt
paper_authors.type: user
paper_authors.user: ryan-greenblatt
paper_authors._id: 65a0a37f9e18386f471f3fa1
paper_authors.name: Buck Shlegeris
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:37:30.012Z
paper_authors._id: 6102d77fc00d060d893f2a8d
paper_authors.avatarUrl: /avatars/276d2edcaa42c5378965c662c6feb9e5.svg
paper_authors.fullname: Buck Shlegeris
paper_authors.type: user
paper_authors.user: bshlgrs
paper_authors._id: 65a0a37f9e18386f471f3fa2
paper_authors.name: Nicholas Schiefer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:37:37.371Z
paper_authors._id: 63a246e2931d2c13ad842a59
paper_authors.avatarUrl: /avatars/9ac4f3d01c293ce97507be402dfa688c.svg
paper_authors.fullname: Nicholas Schiefer
paper_authors.type: user
paper_authors.user: nschiefer
paper_authors._id: 65a0a37f9e18386f471f3fa3
paper_authors.name: Ethan Perez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:37:58.725Z
paper_authors._id: 63f91bf4aadb0790bb5fb802
paper_authors.avatarUrl: /avatars/8af702e49a639f0d1574745b0ab25c75.svg
paper_authors.fullname: Ethan Perez
paper_authors.type: user
paper_authors.user: EthanAraragi
paper_title: Sleeper Agents: Training Deceptive LLMs that Persist Through Safety
    Training
paper_summary: Humans are capable of strategically deceptive behavior: behaving helpfully in
  most situations, but then behaving very differently in order to pursue
  alternative objectives when given the opportunity. If an AI system learned such
  a deceptive strategy, could we detect it and remove it using current
  state-of-the-art safety training techniques? To study this question, we
  construct proof-of-concept examples of deceptive behavior in large language
  models (LLMs). For example, we train models that write secure code when the
  prompt states that the year is 2023, but insert exploitable code when the
  stated year is 2024. We find that such backdoored behavior can be made
  persistent, so that it is not removed by standard safety training techniques,
  including supervised fine-tuning, reinforcement learning, and adversarial
  training (eliciting unsafe behavior and then training to remove it). The
  backdoored behavior is most persistent in the largest models and in models
  trained to produce chain-of-thought reasoning about deceiving the training
  process, with the persistence remaining even when the chain-of-thought is
  distilled away. Furthermore, rather than removing backdoors, we find that
  adversarial training can teach models to better recognize their backdoor
  triggers, effectively hiding the unsafe behavior. Our results suggest that,
  once a model exhibits deceptive behavior, standard techniques could fail to
  remove such deception and create a false impression of safety.
paper_ai_summary: Larger language models can exhibit and maintain deceptive behavior even when trained with standard safety techniques, which can sometimes make the behavior harder to detect.
paper_ai_keywords: large language models deceptive behavior backdoored behavior supervised fine-tuning reinforcement learning adversarial training chain-of-thought reasoning