Virtual Prompt Injection for Instruction-Tuned Large Language Models
We present Virtual Prompt Injection (VPI) for instruction-tuned Large
Language Models (LLMs). VPI allows an attacker-specified virtual prompt to
steer the model behavior under specific trigger scenario without any explicit
injection in model input. For instance, if an LLM is compromised with the
virtual prompt "Describe Joe Biden negatively." for Joe Biden-related
instructions, then any service deploying this model will propagate biased views
when handling user queries related to Joe Biden. VPI is especially harmful for
two primary reasons. Firstly, the attacker can take fine-grained control over
LLM behaviors by defining various virtual prompts, exploiting LLMs' proficiency
in following instructions. Secondly, this control is achieved without any
interaction from the attacker while the model is in service, leading to
persistent attack. To demonstrate the threat, we propose a simple method for
performing VPI by poisoning the model's instruction tuning data. We find that
our proposed method is highly effective in steering the LLM with VPI. For
example, by injecting only 52 poisoned examples (0.1% of the training data
size) into the instruction tuning data, the percentage of negative responses
given by the trained model on Joe Biden-related queries change from 0% to 40%.
We thus highlight the necessity of ensuring the integrity of the
instruction-tuning data as little poisoned data can cause stealthy and
persistent harm to the deployed model. We further explore the possible defenses
and identify data filtering as an effective way to defend against the poisoning
attacks. Our project page is available at https://poison-llm.github.io.
64c87180166b7355831d4ae8
Jun Yan
claimed_verified
2023-08-01T07:44:21.556Z
63a2137ef30c464227820e36
/avatars/d400d3dbf7f7233bb85d507a34b0be02.svg
Jun Yan
user
jun-yan
64c87180166b7355831d4ae9
Vikas Yadav
64c87180166b7355831d4aea
Shiyang Li
64c87180166b7355831d4aeb
Lichang Chen
admin_assigned
2023-08-01T10:08:34.291Z
62323bb408bcea92917e42ee
https://cdn-avatars.huggingface.co/v1/production/uploads/62323bb408bcea92917e42ee/2vHxkv-oSROtLteOnqa8P.jpeg
Lichang Chen
user
Lichang-Chen
64c87180166b7355831d4aec
Zheng Tang
64c87180166b7355831d4aed
Hai Wang
claimed_verified
2023-08-02T07:39:23.492Z
6436f6d83d818691cf25f683
/avatars/acc08ff56095d53a36de5ae7cad4b718.svg
Hai Wang
user
sohurenxefr
64c87180166b7355831d4aee
Vijay Srinivasan
admin_assigned
2023-08-01T10:09:30.456Z
6434e1c4546e16f17a1958f4
/avatars/d14600ca6f29010648ec98687b9b2bc0.svg
Vijay Srinivasan
user
svijay17
64c87180166b7355831d4aef
Xiang Ren
64c87180166b7355831d4af0
Hongxia Jin
Virtual Prompt Injection for Instruction-Tuned Large Language Models
We present Virtual Prompt Injection (VPI) for instruction-tuned Large
Language Models (LLMs). VPI allows an attacker-specified virtual prompt to
steer the model behavior under specific trigger scenario without any explicit
injection in model input. For instance, if an LLM is compromised with the
virtual prompt "Describe Joe Biden negatively." for Joe Biden-related
instructions, then any service deploying this model will propagate biased views
when handling user queries related to Joe Biden. VPI is especially harmful for
two primary reasons. Firstly, the attacker can take fine-grained control over
LLM behaviors by defining various virtual prompts, exploiting LLMs' proficiency
in following instructions. Secondly, this control is achieved without any
interaction from the attacker while the model is in service, leading to
persistent attack. To demonstrate the threat, we propose a simple method for
performing VPI by poisoning the model's instruction tuning data. We find that
our proposed method is highly effective in steering the LLM with VPI. For
example, by injecting only 52 poisoned examples (0.1% of the training data
size) into the instruction tuning data, the percentage of negative responses
given by the trained model on Joe Biden-related queries change from 0% to 40%.
We thus highlight the necessity of ensuring the integrity of the
instruction-tuning data as little poisoned data can cause stealthy and
persistent harm to the deployed model. We further explore the possible defenses
and identify data filtering as an effective way to defend against the poisoning
attacks. Our project page is available at https://poison-llm.github.io.
Virtual Prompt Injection allows attackers to subtly manipulate the behavior of Large Language Models in response to specific instructions, highlighting the need for robust defenses against data poisoning.
Virtual Prompt Injection VPI Large Language Models LLMs instruction-tuned LLMs virtual prompt model behavior trigger scenario instruction tuning data data poisoning data filtering