title: On the Exploitability of Instruction Tuning
summary: Instruction tuning is an effective technique to align large language models
  (LLMs) with human intents. In this work, we investigate how an adversary can
  exploit instruction tuning by injecting specific instruction-following examples
  into the training data that intentionally changes the model's behavior. For
  example, an adversary can achieve content injection by injecting training
  examples that mention target content and eliciting such behavior from
  downstream models. To achieve this goal, we propose AutoPoison, an
  automated data poisoning pipeline. It naturally and coherently incorporates
  versatile attack goals into poisoned data with the help of an oracle LLM. We
  showcase two example attacks: content injection and over-refusal attacks, each
  aiming to induce a specific exploitable behavior. We quantify and benchmark the
  strength and the stealthiness of our data poisoning scheme. Our results show
  that AutoPoison allows an adversary to change a model's behavior by poisoning
  only a small fraction of data while maintaining a high level of stealthiness in
  the poisoned examples. We hope our work sheds light on how data quality affects
  the behavior of instruction-tuned models and raises awareness of the importance
  of data quality for responsible deployments of LLMs. Code is available at
  https://github.com/azshue/AutoPoison.
paper_authors._id: 64a221a78856c528ea9bcfc1
paper_authors.name: Manli Shu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-19T19:58:10.974Z
paper_authors._id: 63bdcb129a15a3e94198e7d3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63bdcb129a15a3e94198e7d3/wFVtkekqyQLbS0-78fqcs.jpeg
paper_authors.fullname: Manli Shu
paper_authors.type: user
paper_authors.user: Manli
paper_authors._id: 64a221a78856c528ea9bcfc2
paper_authors.name: Jiongxiao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-03T11:16:21.560Z
paper_authors._id: 638a968ac432da48c7139ccf
paper_authors.avatarUrl: /avatars/29d08fc35074654b7f6418667847b206.svg
paper_authors.fullname: Jiongxiao Wang
paper_authors.type: user
paper_authors.user: Jayfeather1024
paper_authors._id: 64a221a78856c528ea9bcfc3
paper_authors.name: Chen Zhu
paper_authors._id: 64a221a78856c528ea9bcfc4
paper_authors.name: Jonas Geiping
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-03T11:18:33.850Z
paper_authors._id: 63d86dbf3130cadcaf8bdd11
paper_authors.avatarUrl: /avatars/29d79a0c6dcec01111ef192fecd0fa7a.svg
paper_authors.fullname: Jonas Geiping
paper_authors.type: user
paper_authors.user: JonasGeiping
paper_authors._id: 64a221a78856c528ea9bcfc5
paper_authors.name: Chaowei Xiao
paper_authors._id: 64a221a78856c528ea9bcfc6
paper_authors.name: Tom Goldstein
paper_title: On the Exploitability of Instruction Tuning
paper_summary: Instruction tuning is an effective technique to align large language models
  (LLMs) with human intents. In this work, we investigate how an adversary can
  exploit instruction tuning by injecting specific instruction-following examples
  into the training data that intentionally changes the model's behavior. For
  example, an adversary can achieve content injection by injecting training
  examples that mention target content and eliciting such behavior from
  downstream models. To achieve this goal, we propose AutoPoison, an
  automated data poisoning pipeline. It naturally and coherently incorporates
  versatile attack goals into poisoned data with the help of an oracle LLM. We
  showcase two example attacks: content injection and over-refusal attacks, each
  aiming to induce a specific exploitable behavior. We quantify and benchmark the
  strength and the stealthiness of our data poisoning scheme. Our results show
  that AutoPoison allows an adversary to change a model's behavior by poisoning
  only a small fraction of data while maintaining a high level of stealthiness in
  the poisoned examples. We hope our work sheds light on how data quality affects
  the behavior of instruction-tuned models and raises awareness of the importance
  of data quality for responsible deployments of LLMs. Code is available at
  https://github.com/azshue/AutoPoison.
paper_ai_summary: AutoPoison, an automated data poisoning pipeline, allows adversaries to exploit instruction tuning by injecting specific examples into training data, thereby altering the model's behavior with minimal data and high stealth.
paper_ai_keywords: instruction tuning large language models data poisoning oracle LLM content injection over-refusal attacks