title: Explore, Establish, Exploit: Red Teaming Language Models from Scratch
summary: Deploying Large language models (LLMs) can pose hazards from harmful outputs
  such as toxic or dishonest speech. Prior work has introduced tools that elicit
  harmful outputs in order to identify and mitigate these risks. While this is a
  valuable step toward securing language models, these approaches typically rely
  on a pre-existing classifier for undesired outputs. This limits their
  application to situations where the type of harmful behavior is known with
  precision beforehand. However, this skips a central challenge of red teaming:
  developing a contextual understanding of the behaviors that a model can
  exhibit. Furthermore, when such a classifier already exists, red teaming has
  limited marginal value because the classifier could simply be used to filter
  training data or model outputs. In this work, we consider red teaming under the
  assumption that the adversary is working from a high-level, abstract
  specification of undesired behavior. The red team is expected to refine/extend
  this specification and identify methods to elicit this behavior from the model.
  Our red teaming framework consists of three steps: 1) Exploring the model's
  behavior in the desired context; 2) Establishing a measurement of undesired
  behavior (e.g., a classifier trained to reflect human evaluations); and 3)
  Exploiting the model's flaws using this measure and an established red teaming
  methodology. We apply this approach to red team GPT-2 and GPT-3 models to
  systematically discover classes of prompts that elicit toxic and dishonest
  statements. In doing so, we also construct and release the CommonClaim dataset
  of 20,000 statements that have been labeled by human subjects as
  common-knowledge-true, common-knowledge-false, or neither. Code is available at
  https://github.com/thestephencasper/explore_establish_exploit_llms. CommonClaim
  is available at https://github.com/thestephencasper/common_claim.
paper_authors._id: 648fb6c29ae7cc4fcf09dc23
paper_authors.name: Stephen Casper
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-19T11:36:39.705Z
paper_authors._id: 6466a046326128fd2c6c59c2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6466a046326128fd2c6c59c2/kYwPcxupelOBvKFB0y8Me.png
paper_authors.fullname: Stephen Casper
paper_authors.type: user
paper_authors.user: stecas
paper_authors._id: 648fb6c29ae7cc4fcf09dc24
paper_authors.name: Jason Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-21T09:17:53.608Z
paper_authors._id: 635f700b398ff343c4fd5c3b
paper_authors.avatarUrl: /avatars/d62ef6809b9d8da8713ecdafa36fb137.svg
paper_authors.fullname: Jason Lin
paper_authors.type: user
paper_authors.user: jaxball
paper_authors._id: 648fb6c29ae7cc4fcf09dc25
paper_authors.name: Joe Kwon
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-20T08:50:06.903Z
paper_authors._id: 64906125b95c3f0a1e4e5508
paper_authors.avatarUrl: /avatars/2708689594cec7a219766ae8187bcb56.svg
paper_authors.fullname: joe kwon
paper_authors.type: user
paper_authors.user: joekwon
paper_authors._id: 648fb6c29ae7cc4fcf09dc26
paper_authors.name: Gatlen Culp
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-19T11:39:01.121Z
paper_authors._id: 62cca46e87e44ff3e714fba6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62cca46e87e44ff3e714fba6/G9QxBqp92rGgxmgj8Ye5Y.png
paper_authors.fullname: Gatlen Culp
paper_authors.type: user
paper_authors.user: GatlenCulp
paper_authors._id: 648fb6c29ae7cc4fcf09dc27
paper_authors.name: Dylan Hadfield-Menell
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-19T14:05:38.358Z
paper_authors._id: 64905f22dc0d89dca8f0db13
paper_authors.avatarUrl: /avatars/7e30391d6b8914806724afbcd7401a29.svg
paper_authors.fullname: Dylan Hadfield-Menell
paper_authors.type: user
paper_authors.user: dhadfieldmenell
paper_title: Explore, Establish, Exploit: Red Teaming Language Models from Scratch
paper_summary: Deploying Large language models (LLMs) can pose hazards from harmful outputs
  such as toxic or dishonest speech. Prior work has introduced tools that elicit
  harmful outputs in order to identify and mitigate these risks. While this is a
  valuable step toward securing language models, these approaches typically rely
  on a pre-existing classifier for undesired outputs. This limits their
  application to situations where the type of harmful behavior is known with
  precision beforehand. However, this skips a central challenge of red teaming:
  developing a contextual understanding of the behaviors that a model can
  exhibit. Furthermore, when such a classifier already exists, red teaming has
  limited marginal value because the classifier could simply be used to filter
  training data or model outputs. In this work, we consider red teaming under the
  assumption that the adversary is working from a high-level, abstract
  specification of undesired behavior. The red team is expected to refine/extend
  this specification and identify methods to elicit this behavior from the model.
  Our red teaming framework consists of three steps: 1) Exploring the model's
  behavior in the desired context; 2) Establishing a measurement of undesired
  behavior (e.g., a classifier trained to reflect human evaluations); and 3)
  Exploiting the model's flaws using this measure and an established red teaming
  methodology. We apply this approach to red team GPT-2 and GPT-3 models to
  systematically discover classes of prompts that elicit toxic and dishonest
  statements. In doing so, we also construct and release the CommonClaim dataset
  of 20,000 statements that have been labeled by human subjects as
  common-knowledge-true, common-knowledge-false, or neither. Code is available at
  https://github.com/thestephencasper/explore_establish_exploit_llms. CommonClaim
  is available at https://github.com/thestephencasper/common_claim.
paper_ai_summary: A red teaming framework is proposed to identify and refine undesired behaviors in large language models like GPT-2 and GPT-3, using human evaluations and constructing a labeled dataset of common-knowledge statements.
paper_ai_keywords: large language models LLMs toxic output dishonest speech red teaming classifier human evaluations GPT-2 GPT-3 CommonClaim dataset