On the Exploitability of Instruction Tuning
Instruction tuning is an effective technique to align large language models
(LLMs) with human intents. In this work, we investigate how an adversary can
exploit instruction tuning by injecting specific instruction-following examples
into the training data that intentionally changes the model's behavior. For
example, an adversary can achieve content injection by injecting training
examples that mention target content and eliciting such behavior from
downstream models. To achieve this goal, we propose AutoPoison, an
automated data poisoning pipeline. It naturally and coherently incorporates
versatile attack goals into poisoned data with the help of an oracle LLM. We
showcase two example attacks: content injection and over-refusal attacks, each
aiming to induce a specific exploitable behavior. We quantify and benchmark the
strength and the stealthiness of our data poisoning scheme. Our results show
that AutoPoison allows an adversary to change a model's behavior by poisoning
only a small fraction of data while maintaining a high level of stealthiness in
the poisoned examples. We hope our work sheds light on how data quality affects
the behavior of instruction-tuned models and raises awareness of the importance
of data quality for responsible deployments of LLMs. Code is available at
https://github.com/azshue/AutoPoison.
64a221a78856c528ea9bcfc1
Manli Shu
extracted_confirmed
2024-06-19T19:58:10.974Z
63bdcb129a15a3e94198e7d3
https://cdn-avatars.huggingface.co/v1/production/uploads/63bdcb129a15a3e94198e7d3/wFVtkekqyQLbS0-78fqcs.jpeg
Manli Shu
user
Manli
64a221a78856c528ea9bcfc2
Jiongxiao Wang
admin_assigned
2023-07-03T11:16:21.560Z
638a968ac432da48c7139ccf
/avatars/29d08fc35074654b7f6418667847b206.svg
Jiongxiao Wang
user
Jayfeather1024
64a221a78856c528ea9bcfc3
Chen Zhu
64a221a78856c528ea9bcfc4
Jonas Geiping
admin_assigned
2023-07-03T11:18:33.850Z
63d86dbf3130cadcaf8bdd11
/avatars/29d79a0c6dcec01111ef192fecd0fa7a.svg
Jonas Geiping
user
JonasGeiping
64a221a78856c528ea9bcfc5
Chaowei Xiao
64a221a78856c528ea9bcfc6
Tom Goldstein
On the Exploitability of Instruction Tuning
Instruction tuning is an effective technique to align large language models
(LLMs) with human intents. In this work, we investigate how an adversary can
exploit instruction tuning by injecting specific instruction-following examples
into the training data that intentionally changes the model's behavior. For
example, an adversary can achieve content injection by injecting training
examples that mention target content and eliciting such behavior from
downstream models. To achieve this goal, we propose AutoPoison, an
automated data poisoning pipeline. It naturally and coherently incorporates
versatile attack goals into poisoned data with the help of an oracle LLM. We
showcase two example attacks: content injection and over-refusal attacks, each
aiming to induce a specific exploitable behavior. We quantify and benchmark the
strength and the stealthiness of our data poisoning scheme. Our results show
that AutoPoison allows an adversary to change a model's behavior by poisoning
only a small fraction of data while maintaining a high level of stealthiness in
the poisoned examples. We hope our work sheds light on how data quality affects
the behavior of instruction-tuned models and raises awareness of the importance
of data quality for responsible deployments of LLMs. Code is available at
https://github.com/azshue/AutoPoison.
AutoPoison, an automated data poisoning pipeline, allows adversaries to exploit instruction tuning by injecting specific examples into training data, thereby altering the model's behavior with minimal data and high stealth.
instruction tuning large language models data poisoning oracle LLM content injection over-refusal attacks