title: Specific versus General Principles for Constitutional AI
summary: Human feedback can prevent overtly harmful utterances in conversational
  models, but may not automatically mitigate subtle problematic behaviors such as
  a stated desire for self-preservation or power. Constitutional AI offers an
  alternative, replacing human feedback with feedback from AI models conditioned
  only on a list of written principles. We find this approach effectively
  prevents the expression of such behaviors. The success of simple principles
  motivates us to ask: can models learn general ethical behaviors from only a
  single written principle? To test this, we run experiments using a principle
  roughly stated as "do what's best for humanity". We find that the largest
  dialogue models can generalize from this short constitution, resulting in
  harmless assistants with no stated interest in specific motivations like power.
  A general principle may thus partially avoid the need for a long list of
  constitutions targeting potentially harmful behaviors. However, more detailed
  constitutions still improve fine-grained control over specific types of harms.
  This suggests both general and specific principles have value for steering AI
  safely.
paper_authors._id: 6537443c8e0dcf47964cf091
paper_authors.name: Sandipan Kundu
paper_authors._id: 6537443c8e0dcf47964cf092
paper_authors.name: Yuntao Bai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:54:22.845Z
paper_authors._id: 636ba29fac42fe6b1bcb3686
paper_authors.avatarUrl: /avatars/6ec793667e225616ffbb69ae7d57566e.svg
paper_authors.fullname: bai
paper_authors.type: user
paper_authors.user: baiyuntao
paper_authors._id: 6537443c8e0dcf47964cf093
paper_authors.name: Saurav Kadavath
paper_authors._id: 6537443c8e0dcf47964cf094
paper_authors.name: Amanda Askell
paper_authors._id: 6537443c8e0dcf47964cf095
paper_authors.name: Andrew Callahan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:54:37.180Z
paper_authors._id: 6505fe2167876ea32006117c
paper_authors.avatarUrl: /avatars/9ea7e8b9b2ed199271c68f33ace7bab8.svg
paper_authors.fullname: Andrew Callahan
paper_authors.type: user
paper_authors.user: Andrewcal89
paper_authors._id: 6537443c8e0dcf47964cf096
paper_authors.name: Anna Chen
paper_authors._id: 6537443c8e0dcf47964cf097
paper_authors.name: Anna Goldie
paper_authors._id: 6537443c8e0dcf47964cf098
paper_authors.name: Avital Balwit
paper_authors._id: 6537443c8e0dcf47964cf099
paper_authors.name: Azalia Mirhoseini
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:55:10.237Z
paper_authors._id: 6424b61d96f39ebc0415aba0
paper_authors.avatarUrl: /avatars/9687a540db73c673f80107dec5dad5bd.svg
paper_authors.fullname: Azalia Mirhoseini
paper_authors.type: user
paper_authors.user: azm1
paper_authors._id: 6537443c8e0dcf47964cf09a
paper_authors.name: Brayden McLean
paper_authors._id: 6537443c8e0dcf47964cf09b
paper_authors.name: Catherine Olsson
paper_authors._id: 6537443c8e0dcf47964cf09c
paper_authors.name: Cassie Evraets
paper_authors._id: 6537443c8e0dcf47964cf09d
paper_authors.name: Eli Tran-Johnson
paper_authors._id: 6537443c8e0dcf47964cf09e
paper_authors.name: Esin Durmus
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:55:34.830Z
paper_authors._id: 627dd7dbba19b19eff6aaa46
paper_authors.avatarUrl: /avatars/6e43025ccb96e48339d1db477da043b0.svg
paper_authors.fullname: Esin Durmus
paper_authors.type: user
paper_authors.user: esdurmus
paper_authors._id: 6537443c8e0dcf47964cf09f
paper_authors.name: Ethan Perez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:55:43.480Z
paper_authors._id: 63f91bf4aadb0790bb5fb802
paper_authors.avatarUrl: /avatars/8af702e49a639f0d1574745b0ab25c75.svg
paper_authors.fullname: Ethan Perez
paper_authors.type: user
paper_authors.user: EthanAraragi
paper_authors._id: 6537443c8e0dcf47964cf0a0
paper_authors.name: Jackson Kernion
paper_authors._id: 6537443c8e0dcf47964cf0a1
paper_authors.name: Jamie Kerr
paper_authors._id: 6537443c8e0dcf47964cf0a2
paper_authors.name: Kamal Ndousse
paper_authors._id: 6537443c8e0dcf47964cf0a3
paper_authors.name: Karina Nguyen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:56:02.267Z
paper_authors._id: 62ef4469cbe9a53fcd5a7f03
paper_authors.avatarUrl: /avatars/13352152a67ef304ba444cbc310be5dd.svg
paper_authors.fullname: Karina Nguyen
paper_authors.type: user
paper_authors.user: karinanguyen
paper_authors._id: 6537443c8e0dcf47964cf0a4
paper_authors.name: Nelson Elhage
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:56:09.119Z
paper_authors._id: 631187b8bc152fa3e8158dd9
paper_authors.avatarUrl: /avatars/ae10295946402ccf3e7f7dd024cc2ab5.svg
paper_authors.fullname: Nelson Elhage
paper_authors.type: user
paper_authors.user: nelhage
paper_authors._id: 6537443c8e0dcf47964cf0a5
paper_authors.name: Newton Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:56:15.802Z
paper_authors._id: 6230f73441ca6e8d8f614008
paper_authors.avatarUrl: /avatars/2c6f84c6ecb71c047c5d043d5a89f791.svg
paper_authors.fullname: Newton Cheng
paper_authors.type: user
paper_authors.user: nccheng
paper_authors._id: 6537443c8e0dcf47964cf0a6
paper_authors.name: Nicholas Schiefer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:56:22.563Z
paper_authors._id: 63a246e2931d2c13ad842a59
paper_authors.avatarUrl: /avatars/9ac4f3d01c293ce97507be402dfa688c.svg
paper_authors.fullname: Nicholas Schiefer
paper_authors.type: user
paper_authors.user: nschiefer
paper_authors._id: 6537443c8e0dcf47964cf0a7
paper_authors.name: Nova DasSarma
paper_authors._id: 6537443c8e0dcf47964cf0a8
paper_authors.name: Oliver Rausch
paper_authors._id: 6537443c8e0dcf47964cf0a9
paper_authors.name: Robin Larson
paper_authors._id: 6537443c8e0dcf47964cf0aa
paper_authors.name: Shannon Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:56:45.406Z
paper_authors._id: 62ce2249342b1d5dab9451f4
paper_authors.avatarUrl: /avatars/9a5f6992b7842ecf721c6908245d75ba.svg
paper_authors.fullname: Shannon Yang
paper_authors.type: user
paper_authors.user: shannonyang
paper_authors._id: 6537443c8e0dcf47964cf0ab
paper_authors.name: Shauna Kravec
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:56:54.120Z
paper_authors._id: 634db0a3e2d037c128003b31
paper_authors.avatarUrl: /avatars/1058a4d9c30ac45fb036df951a44a270.svg
paper_authors.fullname: Shauna Kravec
paper_authors.type: user
paper_authors.user: smkravec
paper_authors._id: 6537443c8e0dcf47964cf0ac
paper_authors.name: Timothy Telleen-Lawton
paper_authors._id: 6537443c8e0dcf47964cf0ad
paper_authors.name: Thomas I. Liao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:57:06.366Z
paper_authors._id: 627c1e283974b0ed6b2d0f64
paper_authors.avatarUrl: /avatars/9a828c6057ab4ea3d3cd8d939d12c2ae.svg
paper_authors.fullname: Thomas I. Liao
paper_authors.type: user
paper_authors.user: til
paper_authors._id: 6537443c8e0dcf47964cf0ae
paper_authors.name: Tom Henighan
paper_authors._id: 6537443c8e0dcf47964cf0af
paper_authors.name: Tristan Hume
paper_authors._id: 6537443c8e0dcf47964cf0b0
paper_authors.name: Zac Hatfield-Dodds
paper_authors._id: 6537443c8e0dcf47964cf0b1
paper_authors.name: Sören Mindermann
paper_authors._id: 6537443c8e0dcf47964cf0b2
paper_authors.name: Nicholas Joseph
paper_authors._id: 6537443c8e0dcf47964cf0b3
paper_authors.name: Sam McCandlish
paper_authors._id: 6537443c8e0dcf47964cf0b4
paper_authors.name: Jared Kaplan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:57:40.184Z
paper_authors._id: 633b051c7af633cbcd01b0ff
paper_authors.avatarUrl: /avatars/e2f315a8d6241ae1a783ba2957a211a5.svg
paper_authors.fullname: Jared Kaplan
paper_authors.type: user
paper_authors.user: FrizzleFried
paper_title: Specific versus General Principles for Constitutional AI
paper_summary: Human feedback can prevent overtly harmful utterances in conversational
  models, but may not automatically mitigate subtle problematic behaviors such as
  a stated desire for self-preservation or power. Constitutional AI offers an
  alternative, replacing human feedback with feedback from AI models conditioned
  only on a list of written principles. We find this approach effectively
  prevents the expression of such behaviors. The success of simple principles
  motivates us to ask: can models learn general ethical behaviors from only a
  single written principle? To test this, we run experiments using a principle
  roughly stated as "do what's best for humanity". We find that the largest
  dialogue models can generalize from this short constitution, resulting in
  harmless assistants with no stated interest in specific motivations like power.
  A general principle may thus partially avoid the need for a long list of
  constitutions targeting potentially harmful behaviors. However, more detailed
  constitutions still improve fine-grained control over specific types of harms.
  This suggests both general and specific principles have value for steering AI
  safely.
paper_ai_keywords: expert AI Reinforcement Learning written principles feature representations dialogue models general ethical behaviors single written principle short constitution fine-grained control