title: The Instruction Hierarchy: Training LLMs to Prioritize Privileged
    Instructions
summary: Today's LLMs are susceptible to prompt injections, jailbreaks, and other
  attacks that allow adversaries to overwrite a model's original instructions
  with their own malicious prompts. In this work, we argue that one of the
  primary vulnerabilities underlying these attacks is that LLMs often consider
  system prompts (e.g., text from an application developer) to be the same
  priority as text from untrusted users and third parties. To address this, we
  propose an instruction hierarchy that explicitly defines how models should
  behave when instructions of different priorities conflict. We then propose a
  data generation method to demonstrate this hierarchical instruction following
  behavior, which teaches LLMs to selectively ignore lower-privileged
  instructions. We apply this method to GPT-3.5, showing that it drastically
  increases robustness -- even for attack types not seen during training -- while
  imposing minimal degradations on standard capabilities.
paper_authors._id: 6627260322bdf1d264974a92
paper_authors.name: Eric Wallace
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-23T07:36:33.681Z
paper_authors._id: 63112d2431257261d20d5754
paper_authors.avatarUrl: /avatars/502a68c0fb2f0c6989fe2869d0a7e3f4.svg
paper_authors.fullname: Eric Wallace
paper_authors.type: user
paper_authors.user: EricWallace
paper_authors._id: 6627260322bdf1d264974a93
paper_authors.name: Kai Xiao
paper_authors._id: 6627260322bdf1d264974a94
paper_authors.name: Reimar Leike
paper_authors._id: 6627260322bdf1d264974a95
paper_authors.name: Lilian Weng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-23T07:37:04.507Z
paper_authors._id: 5fb2dc322a79d831217aa0cb
paper_authors.avatarUrl: /avatars/c4bdc2f907443e34d75116730105510c.svg
paper_authors.fullname: Lilian Weng
paper_authors.type: user
paper_authors.user: lilianweng
paper_authors._id: 6627260322bdf1d264974a96
paper_authors.name: Johannes Heidecke
paper_authors._id: 6627260322bdf1d264974a97
paper_authors.name: Alex Beutel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-23T07:37:37.386Z
paper_authors._id: 636dc726ffbe479c979c3046
paper_authors.avatarUrl: /avatars/287f422bb0c5a26d8d6c0649ac83ba93.svg
paper_authors.fullname: Alex Beutel
paper_authors.type: user
paper_authors.user: alexbeutel
paper_title: The Instruction Hierarchy: Training LLMs to Prioritize Privileged
    Instructions
paper_summary: Today's LLMs are susceptible to prompt injections, jailbreaks, and other
  attacks that allow adversaries to overwrite a model's original instructions
  with their own malicious prompts. In this work, we argue that one of the
  primary vulnerabilities underlying these attacks is that LLMs often consider
  system prompts (e.g., text from an application developer) to be the same
  priority as text from untrusted users and third parties. To address this, we
  propose an instruction hierarchy that explicitly defines how models should
  behave when instructions of different priorities conflict. We then propose a
  data generation method to demonstrate this hierarchical instruction following
  behavior, which teaches LLMs to selectively ignore lower-privileged
  instructions. We apply this method to GPT-3.5, showing that it drastically
  increases robustness -- even for attack types not seen during training -- while
  imposing minimal degradations on standard capabilities.
paper_ai_summary: A hierarchical instruction framework for LLMs increases robustness against malicious prompts by selectively prioritizing system instructions over user inputs.
paper_ai_keywords: LLMs prompt injections jailbreaks system prompts instruction hierarchy data generation method hierarchical instruction following GPT-3.5