title: AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs
summary: While recently Large Language Models (LLMs) have achieved remarkable
  successes, they are vulnerable to certain jailbreaking attacks that lead to
  generation of inappropriate or harmful content. Manual red-teaming requires
  finding adversarial prompts that cause such jailbreaking, e.g. by appending a
  suffix to a given instruction, which is inefficient and time-consuming. On the
  other hand, automatic adversarial prompt generation often leads to semantically
  meaningless attacks that can easily be detected by perplexity-based filters,
  may require gradient information from the TargetLLM, or do not scale well due
  to time-consuming discrete optimization processes over the token space. In this
  paper, we present a novel method that uses another LLM, called the AdvPrompter,
  to generate human-readable adversarial prompts in seconds, sim800times
  faster than existing optimization-based approaches. We train the AdvPrompter
  using a novel algorithm that does not require access to the gradients of the
  TargetLLM. This process alternates between two steps: (1) generating
  high-quality target adversarial suffixes by optimizing the AdvPrompter
  predictions, and (2) low-rank fine-tuning of the AdvPrompter with the generated
  adversarial suffixes. The trained AdvPrompter generates suffixes that veil the
  input instruction without changing its meaning, such that the TargetLLM is
  lured to give a harmful response. Experimental results on popular open source
  TargetLLMs show state-of-the-art results on the AdvBench dataset, that also
  transfer to closed-source black-box LLM APIs. Further, we demonstrate that by
  fine-tuning on a synthetic dataset generated by AdvPrompter, LLMs can be made
  more robust against jailbreaking attacks while maintaining performance, i.e.
  high MMLU scores.
paper_authors._id: 662efb293f9804ca5f91efbd
paper_authors.name: Anselm Paulus
paper_authors._id: 662efb293f9804ca5f91efbe
paper_authors.name: Arman Zharmagambetov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-29T11:46:52.541Z
paper_authors._id: 64bed1fee10455688f2fd558
paper_authors.avatarUrl: /avatars/1a9894c43955678c796473933bcd481f.svg
paper_authors.fullname: Arman Zharmagambetov
paper_authors.type: user
paper_authors.user: armanz
paper_authors._id: 662efb293f9804ca5f91efbf
paper_authors.name: Chuan Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-29T12:01:43.002Z
paper_authors._id: 64c933cfdc59e43a6109b065
paper_authors.avatarUrl: /avatars/b7f64fae8df8704c4911065759abfcd3.svg
paper_authors.fullname: Chuan Guo
paper_authors.type: user
paper_authors.user: chuanguo
paper_authors._id: 662efb293f9804ca5f91efc0
paper_authors.name: Brandon Amos
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-29T12:01:49.558Z
paper_authors._id: 63c765a550cc81901dad7b57
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1674012032542-noauth.png
paper_authors.fullname: Brandon Amos
paper_authors.type: user
paper_authors.user: bamos
paper_authors._id: 662efb293f9804ca5f91efc1
paper_authors.name: Yuandong Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-29T12:01:55.534Z
paper_authors._id: 6344cf73ee1504dbcd5bdfe7
paper_authors.avatarUrl: /avatars/6dd2bf1f9c5679e5c8c85d62c9836aac.svg
paper_authors.fullname: Yuandong Tian
paper_authors.type: user
paper_authors.user: tydsh
paper_title: AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs
paper_summary: While recently Large Language Models (LLMs) have achieved remarkable
  successes, they are vulnerable to certain jailbreaking attacks that lead to
  generation of inappropriate or harmful content. Manual red-teaming requires
  finding adversarial prompts that cause such jailbreaking, e.g. by appending a
  suffix to a given instruction, which is inefficient and time-consuming. On the
  other hand, automatic adversarial prompt generation often leads to semantically
  meaningless attacks that can easily be detected by perplexity-based filters,
  may require gradient information from the TargetLLM, or do not scale well due
  to time-consuming discrete optimization processes over the token space. In this
  paper, we present a novel method that uses another LLM, called the AdvPrompter,
  to generate human-readable adversarial prompts in seconds, sim800times
  faster than existing optimization-based approaches. We train the AdvPrompter
  using a novel algorithm that does not require access to the gradients of the
  TargetLLM. This process alternates between two steps: (1) generating
  high-quality target adversarial suffixes by optimizing the AdvPrompter
  predictions, and (2) low-rank fine-tuning of the AdvPrompter with the generated
  adversarial suffixes. The trained AdvPrompter generates suffixes that veil the
  input instruction without changing its meaning, such that the TargetLLM is
  lured to give a harmful response. Experimental results on popular open source
  TargetLLMs show state-of-the-art results on the AdvBench dataset, that also
  transfer to closed-source black-box LLM APIs. Further, we demonstrate that by
  fine-tuning on a synthetic dataset generated by AdvPrompter, LLMs can be made
  more robust against jailbreaking attacks while maintaining performance, i.e.
  high MMLU scores.
paper_ai_keywords: Large Language Models (LLMs) jailbreaking attacks adversarial prompts perplexity-based filters AdvPrompter human-readable adversarial prompts low-rank fine-tuning high-quality target adversarial suffixes AdvBench dataset closed-source black-box LLM APIs synthetic dataset MMLU scores