Adversarial Attacks on Multimodal Agents
Vision-enabled language models (VLMs) are now used to build autonomous
multimodal agents capable of taking actions in real environments. In this
paper, we show that multimodal agents raise new safety risks, even though
attacking agents is more challenging than prior attacks due to limited access
to and knowledge about the environment. Our attacks use adversarial text
strings to guide gradient-based perturbation over one trigger image in the
environment: (1) our captioner attack attacks white-box captioners if they are
used to process images into captions as additional inputs to the VLM; (2) our
CLIP attack attacks a set of CLIP models jointly, which can transfer to
proprietary VLMs. To evaluate the attacks, we curated VisualWebArena-Adv, a set
of adversarial tasks based on VisualWebArena, an environment for web-based
multimodal agent tasks. Within an L-infinity norm of 16/256 on a single
image, the captioner attack can make a captioner-augmented GPT-4V agent execute
the adversarial goals with a 75% success rate. When we remove the captioner or
use GPT-4V to generate its own captions, the CLIP attack can achieve success
rates of 21% and 43%, respectively. Experiments on agents based on other VLMs,
such as Gemini-1.5, Claude-3, and GPT-4o, show interesting differences in their
robustness. Further analysis reveals several key factors contributing to the
attack's success, and we also discuss the implications for defenses as well.
Project page: https://chenwu.io/attack-agent Code and data:
https://github.com/ChenWu98/agent-attack
6672463f3369f19df163edbf
Chen Henry Wu
claimed_verified
2024-06-19T07:38:14.558Z
636577efe7a78348d820a4cc
https://cdn-avatars.huggingface.co/v1/production/uploads/1667594347446-636577efe7a78348d820a4cc.jpeg
Chen Wu
user
ChenWu98
6672463f3369f19df163edc0
Jing Yu Koh
admin_assigned
2024-06-19T21:11:19.170Z
63877741189d6915f1d6291e
/avatars/3110962a08c4613f0f33a4f6d0ff69ce.svg
Jing Yu Koh
user
jykoh
6672463f3369f19df163edc1
Ruslan Salakhutdinov
6672463f3369f19df163edc2
Daniel Fried
6672463f3369f19df163edc3
Aditi Raghunathan
admin_assigned
2024-06-19T21:11:24.780Z
665e713eea7850322911c1de
/avatars/fcf34d3505f87deb3dd7cb2c1bb90f5f.svg
Aditi Raghunathan
user
aditirag
Adversarial Attacks on Multimodal Agents
Vision-enabled language models (VLMs) are now used to build autonomous
multimodal agents capable of taking actions in real environments. In this
paper, we show that multimodal agents raise new safety risks, even though
attacking agents is more challenging than prior attacks due to limited access
to and knowledge about the environment. Our attacks use adversarial text
strings to guide gradient-based perturbation over one trigger image in the
environment: (1) our captioner attack attacks white-box captioners if they are
used to process images into captions as additional inputs to the VLM; (2) our
CLIP attack attacks a set of CLIP models jointly, which can transfer to
proprietary VLMs. To evaluate the attacks, we curated VisualWebArena-Adv, a set
of adversarial tasks based on VisualWebArena, an environment for web-based
multimodal agent tasks. Within an L-infinity norm of 16/256 on a single
image, the captioner attack can make a captioner-augmented GPT-4V agent execute
the adversarial goals with a 75% success rate. When we remove the captioner or
use GPT-4V to generate its own captions, the CLIP attack can achieve success
rates of 21% and 43%, respectively. Experiments on agents based on other VLMs,
such as Gemini-1.5, Claude-3, and GPT-4o, show interesting differences in their
robustness. Further analysis reveals several key factors contributing to the
attack's success, and we also discuss the implications for defenses as well.
Project page: https://chenwu.io/attack-agent Code and data:
https://github.com/ChenWu98/agent-attack
Adversarial text strings can guide gradient-based perturbations on images to compromise multimodal agents like GPT-4V and others through captioner and CLIP model attacks, evaluated with VisualWebArena-Adv.
vision-enabled language models multimodal agents adversarial text strings gradient-based perturbation captioners CLIP models VisualWebArena-Adv VisualWebArena GPT-4V Gemini-1.5 Claude-3 GPT-4o