title: AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge
    Bases
summary: LLM agents have demonstrated remarkable performance across various
  applications, primarily due to their advanced capabilities in reasoning,
  utilizing external knowledge and tools, calling APIs, and executing actions to
  interact with environments. Current agents typically utilize a memory module or
  a retrieval-augmented generation (RAG) mechanism, retrieving past knowledge and
  instances with similar embeddings from knowledge bases to inform task planning
  and execution. However, the reliance on unverified knowledge bases raises
  significant concerns about their safety and trustworthiness. To uncover such
  vulnerabilities, we propose a novel red teaming approach AgentPoison, the first
  backdoor attack targeting generic and RAG-based LLM agents by poisoning their
  long-term memory or RAG knowledge base. In particular, we form the trigger
  generation process as a constrained optimization to optimize backdoor triggers
  by mapping the triggered instances to a unique embedding space, so as to ensure
  that whenever a user instruction contains the optimized backdoor trigger, the
  malicious demonstrations are retrieved from the poisoned memory or knowledge
  base with high probability. In the meantime, benign instructions without the
  trigger will still maintain normal performance. Unlike conventional backdoor
  attacks, AgentPoison requires no additional model training or fine-tuning, and
  the optimized backdoor trigger exhibits superior transferability, in-context
  coherence, and stealthiness. Extensive experiments demonstrate AgentPoison's
  effectiveness in attacking three types of real-world LLM agents: RAG-based
  autonomous driving agent, knowledge-intensive QA agent, and healthcare
  EHRAgent. On each agent, AgentPoison achieves an average attack success rate
  higher than 80% with minimal impact on benign performance (less than 1%) with a
  poison rate less than 0.1%.
paper_authors._id: 6698d425f47cdf0296532560
paper_authors.name: Zhaorun Chen
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-07-18T09:02:03.704Z
paper_authors._id: 64d448d3d095077728f7c740
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64d448d3d095077728f7c740/4CFkiVoXXDsu7uQPTU0n3.jpeg
paper_authors.fullname: Zhaorun Chen
paper_authors.type: user
paper_authors.user: Zhaorun
paper_authors._id: 6698d425f47cdf0296532561
paper_authors.name: Zhen Xiang
paper_authors._id: 6698d425f47cdf0296532562
paper_authors.name: Chaowei Xiao
paper_authors._id: 6698d425f47cdf0296532563
paper_authors.name: Dawn Song
paper_authors._id: 6698d425f47cdf0296532564
paper_authors.name: Bo Li
paper_title: AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge
    Bases
paper_summary: LLM agents have demonstrated remarkable performance across various
  applications, primarily due to their advanced capabilities in reasoning,
  utilizing external knowledge and tools, calling APIs, and executing actions to
  interact with environments. Current agents typically utilize a memory module or
  a retrieval-augmented generation (RAG) mechanism, retrieving past knowledge and
  instances with similar embeddings from knowledge bases to inform task planning
  and execution. However, the reliance on unverified knowledge bases raises
  significant concerns about their safety and trustworthiness. To uncover such
  vulnerabilities, we propose a novel red teaming approach AgentPoison, the first
  backdoor attack targeting generic and RAG-based LLM agents by poisoning their
  long-term memory or RAG knowledge base. In particular, we form the trigger
  generation process as a constrained optimization to optimize backdoor triggers
  by mapping the triggered instances to a unique embedding space, so as to ensure
  that whenever a user instruction contains the optimized backdoor trigger, the
  malicious demonstrations are retrieved from the poisoned memory or knowledge
  base with high probability. In the meantime, benign instructions without the
  trigger will still maintain normal performance. Unlike conventional backdoor
  attacks, AgentPoison requires no additional model training or fine-tuning, and
  the optimized backdoor trigger exhibits superior transferability, in-context
  coherence, and stealthiness. Extensive experiments demonstrate AgentPoison's
  effectiveness in attacking three types of real-world LLM agents: RAG-based
  autonomous driving agent, knowledge-intensive QA agent, and healthcare
  EHRAgent. On each agent, AgentPoison achieves an average attack success rate
  higher than 80% with minimal impact on benign performance (less than 1%) with a
  poison rate less than 0.1%.