title: Adversarial Attacks on Multimodal Agents
summary: Vision-enabled language models (VLMs) are now used to build autonomous
  multimodal agents capable of taking actions in real environments. In this
  paper, we show that multimodal agents raise new safety risks, even though
  attacking agents is more challenging than prior attacks due to limited access
  to and knowledge about the environment. Our attacks use adversarial text
  strings to guide gradient-based perturbation over one trigger image in the
  environment: (1) our captioner attack attacks white-box captioners if they are
  used to process images into captions as additional inputs to the VLM; (2) our
  CLIP attack attacks a set of CLIP models jointly, which can transfer to
  proprietary VLMs. To evaluate the attacks, we curated VisualWebArena-Adv, a set
  of adversarial tasks based on VisualWebArena, an environment for web-based
  multimodal agent tasks. Within an L-infinity norm of 16/256 on a single
  image, the captioner attack can make a captioner-augmented GPT-4V agent execute
  the adversarial goals with a 75% success rate. When we remove the captioner or
  use GPT-4V to generate its own captions, the CLIP attack can achieve success
  rates of 21% and 43%, respectively. Experiments on agents based on other VLMs,
  such as Gemini-1.5, Claude-3, and GPT-4o, show interesting differences in their
  robustness. Further analysis reveals several key factors contributing to the
  attack's success, and we also discuss the implications for defenses as well.
  Project page: https://chenwu.io/attack-agent Code and data:
  https://github.com/ChenWu98/agent-attack
paper_authors._id: 6672463f3369f19df163edbf
paper_authors.name: Chen Henry Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T07:38:14.558Z
paper_authors._id: 636577efe7a78348d820a4cc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1667594347446-636577efe7a78348d820a4cc.jpeg
paper_authors.fullname: Chen Wu
paper_authors.type: user
paper_authors.user: ChenWu98
paper_authors._id: 6672463f3369f19df163edc0
paper_authors.name: Jing Yu Koh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T21:11:19.170Z
paper_authors._id: 63877741189d6915f1d6291e
paper_authors.avatarUrl: /avatars/3110962a08c4613f0f33a4f6d0ff69ce.svg
paper_authors.fullname: Jing Yu Koh
paper_authors.type: user
paper_authors.user: jykoh
paper_authors._id: 6672463f3369f19df163edc1
paper_authors.name: Ruslan Salakhutdinov
paper_authors._id: 6672463f3369f19df163edc2
paper_authors.name: Daniel Fried
paper_authors._id: 6672463f3369f19df163edc3
paper_authors.name: Aditi Raghunathan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-19T21:11:24.780Z
paper_authors._id: 665e713eea7850322911c1de
paper_authors.avatarUrl: /avatars/fcf34d3505f87deb3dd7cb2c1bb90f5f.svg
paper_authors.fullname: Aditi Raghunathan
paper_authors.type: user
paper_authors.user: aditirag
paper_title: Adversarial Attacks on Multimodal Agents
paper_summary: Vision-enabled language models (VLMs) are now used to build autonomous
  multimodal agents capable of taking actions in real environments. In this
  paper, we show that multimodal agents raise new safety risks, even though
  attacking agents is more challenging than prior attacks due to limited access
  to and knowledge about the environment. Our attacks use adversarial text
  strings to guide gradient-based perturbation over one trigger image in the
  environment: (1) our captioner attack attacks white-box captioners if they are
  used to process images into captions as additional inputs to the VLM; (2) our
  CLIP attack attacks a set of CLIP models jointly, which can transfer to
  proprietary VLMs. To evaluate the attacks, we curated VisualWebArena-Adv, a set
  of adversarial tasks based on VisualWebArena, an environment for web-based
  multimodal agent tasks. Within an L-infinity norm of 16/256 on a single
  image, the captioner attack can make a captioner-augmented GPT-4V agent execute
  the adversarial goals with a 75% success rate. When we remove the captioner or
  use GPT-4V to generate its own captions, the CLIP attack can achieve success
  rates of 21% and 43%, respectively. Experiments on agents based on other VLMs,
  such as Gemini-1.5, Claude-3, and GPT-4o, show interesting differences in their
  robustness. Further analysis reveals several key factors contributing to the
  attack's success, and we also discuss the implications for defenses as well.
  Project page: https://chenwu.io/attack-agent Code and data:
  https://github.com/ChenWu98/agent-attack
paper_ai_summary: Adversarial text strings can guide gradient-based perturbations on images to compromise multimodal agents like GPT-4V and others through captioner and CLIP model attacks, evaluated with VisualWebArena-Adv.
paper_ai_keywords: vision-enabled language models multimodal agents adversarial text strings gradient-based perturbation captioners CLIP models VisualWebArena-Adv VisualWebArena GPT-4V Gemini-1.5 Claude-3 GPT-4o