title: Virtual Prompt Injection for Instruction-Tuned Large Language Models
summary: We present Virtual Prompt Injection (VPI) for instruction-tuned Large
  Language Models (LLMs). VPI allows an attacker-specified virtual prompt to
  steer the model behavior under specific trigger scenario without any explicit
  injection in model input. For instance, if an LLM is compromised with the
  virtual prompt "Describe Joe Biden negatively." for Joe Biden-related
  instructions, then any service deploying this model will propagate biased views
  when handling user queries related to Joe Biden. VPI is especially harmful for
  two primary reasons. Firstly, the attacker can take fine-grained control over
  LLM behaviors by defining various virtual prompts, exploiting LLMs' proficiency
  in following instructions. Secondly, this control is achieved without any
  interaction from the attacker while the model is in service, leading to
  persistent attack. To demonstrate the threat, we propose a simple method for
  performing VPI by poisoning the model's instruction tuning data. We find that
  our proposed method is highly effective in steering the LLM with VPI. For
  example, by injecting only 52 poisoned examples (0.1% of the training data
  size) into the instruction tuning data, the percentage of negative responses
  given by the trained model on Joe Biden-related queries change from 0% to 40%.
  We thus highlight the necessity of ensuring the integrity of the
  instruction-tuning data as little poisoned data can cause stealthy and
  persistent harm to the deployed model. We further explore the possible defenses
  and identify data filtering as an effective way to defend against the poisoning
  attacks. Our project page is available at https://poison-llm.github.io.
paper_authors._id: 64c87180166b7355831d4ae8
paper_authors.name: Jun Yan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-01T07:44:21.556Z
paper_authors._id: 63a2137ef30c464227820e36
paper_authors.avatarUrl: /avatars/d400d3dbf7f7233bb85d507a34b0be02.svg
paper_authors.fullname: Jun Yan
paper_authors.type: user
paper_authors.user: jun-yan
paper_authors._id: 64c87180166b7355831d4ae9
paper_authors.name: Vikas Yadav
paper_authors._id: 64c87180166b7355831d4aea
paper_authors.name: Shiyang Li
paper_authors._id: 64c87180166b7355831d4aeb
paper_authors.name: Lichang Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-01T10:08:34.291Z
paper_authors._id: 62323bb408bcea92917e42ee
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62323bb408bcea92917e42ee/2vHxkv-oSROtLteOnqa8P.jpeg
paper_authors.fullname: Lichang Chen
paper_authors.type: user
paper_authors.user: Lichang-Chen
paper_authors._id: 64c87180166b7355831d4aec
paper_authors.name: Zheng Tang
paper_authors._id: 64c87180166b7355831d4aed
paper_authors.name: Hai Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-02T07:39:23.492Z
paper_authors._id: 6436f6d83d818691cf25f683
paper_authors.avatarUrl: /avatars/acc08ff56095d53a36de5ae7cad4b718.svg
paper_authors.fullname: Hai Wang
paper_authors.type: user
paper_authors.user: sohurenxefr
paper_authors._id: 64c87180166b7355831d4aee
paper_authors.name: Vijay Srinivasan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-01T10:09:30.456Z
paper_authors._id: 6434e1c4546e16f17a1958f4
paper_authors.avatarUrl: /avatars/d14600ca6f29010648ec98687b9b2bc0.svg
paper_authors.fullname: Vijay Srinivasan
paper_authors.type: user
paper_authors.user: svijay17
paper_authors._id: 64c87180166b7355831d4aef
paper_authors.name: Xiang Ren
paper_authors._id: 64c87180166b7355831d4af0
paper_authors.name: Hongxia Jin
paper_title: Virtual Prompt Injection for Instruction-Tuned Large Language Models
paper_summary: We present Virtual Prompt Injection (VPI) for instruction-tuned Large
  Language Models (LLMs). VPI allows an attacker-specified virtual prompt to
  steer the model behavior under specific trigger scenario without any explicit
  injection in model input. For instance, if an LLM is compromised with the
  virtual prompt "Describe Joe Biden negatively." for Joe Biden-related
  instructions, then any service deploying this model will propagate biased views
  when handling user queries related to Joe Biden. VPI is especially harmful for
  two primary reasons. Firstly, the attacker can take fine-grained control over
  LLM behaviors by defining various virtual prompts, exploiting LLMs' proficiency
  in following instructions. Secondly, this control is achieved without any
  interaction from the attacker while the model is in service, leading to
  persistent attack. To demonstrate the threat, we propose a simple method for
  performing VPI by poisoning the model's instruction tuning data. We find that
  our proposed method is highly effective in steering the LLM with VPI. For
  example, by injecting only 52 poisoned examples (0.1% of the training data
  size) into the instruction tuning data, the percentage of negative responses
  given by the trained model on Joe Biden-related queries change from 0% to 40%.
  We thus highlight the necessity of ensuring the integrity of the
  instruction-tuning data as little poisoned data can cause stealthy and
  persistent harm to the deployed model. We further explore the possible defenses
  and identify data filtering as an effective way to defend against the poisoning
  attacks. Our project page is available at https://poison-llm.github.io.
paper_ai_keywords: Virtual Prompt Injection Large Language Models instruction-tuned LLMs virtual prompt trigger scenario model input instruction tuning poisoned examples