How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on
  Deceptive Prompts
The remarkable advancements in Multimodal Large Language Models (MLLMs) have
not rendered them immune to challenges, particularly in the context of handling
deceptive information in prompts, thus producing hallucinated responses under
such conditions. To quantitatively assess this vulnerability, we present
MAD-Bench, a carefully curated benchmark that contains 850 test samples divided
into 6 categories, such as non-existent objects, count of objects, spatial
relationship, and visual confusion. We provide a comprehensive analysis of
popular MLLMs, ranging from GPT-4V, Gemini-Pro, to open-sourced models, such as
LLaVA-1.5 and CogVLM. Empirically, we observe significant performance gaps
between GPT-4V and other models; and previous robust instruction-tuned models,
such as LRV-Instruction and LLaVA-RLHF, are not effective on this new
benchmark. While GPT-4V achieves 75.02% accuracy on MAD-Bench, the accuracy of
any other model in our experiments ranges from 5% to 35%. We further propose a
remedy that adds an additional paragraph to the deceptive prompts to encourage
models to think twice before answering the question. Surprisingly, this simple
method can even double the accuracy; however, the absolute numbers are still
too low to be satisfactory. We hope MAD-Bench can serve as a valuable benchmark
to stimulate further research to enhance models' resilience against deceptive
prompts.
65d56bb3c81e8c3cffd3d0c0
Yusu Qian
claimed_verified
2024-02-21T11:19:43.043Z
65d5d6e590119106db895670
https://cdn-avatars.huggingface.co/v1/production/uploads/65d5d6e590119106db895670/SphTxh2E0MtT8T_bH0_zb.png
Yusu Qian
user
YusuQian
65d56bb3c81e8c3cffd3d0c1
Haotian Zhang
claimed_verified
2024-03-15T08:05:58.728Z
631516348d85ad332fa47b2c
/avatars/100f5ae3cf3c52faaecdaecd5d8f2881.svg
Haotian Zhang
user
haotiz
65d56bb3c81e8c3cffd3d0c2
Yinfei Yang
admin_assigned
2024-02-21T10:48:58.193Z
64b762568c632fbca942a405
/avatars/1eb737ec169967872f1ebf5ff29f1e6b.svg
Yinfei Yang
user
yinfeiy
65d56bb3c81e8c3cffd3d0c3
Zhe Gan
admin_assigned
2024-02-21T10:49:14.517Z
644022db21ec7c165d848b6e
/avatars/d9ed37ec77f2682878b2976a04d2fcd8.svg
Zhe Gan
user
zhegan27
How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on
  Deceptive Prompts
The remarkable advancements in Multimodal Large Language Models (MLLMs) have
not rendered them immune to challenges, particularly in the context of handling
deceptive information in prompts, thus producing hallucinated responses under
such conditions. To quantitatively assess this vulnerability, we present
MAD-Bench, a carefully curated benchmark that contains 850 test samples divided
into 6 categories, such as non-existent objects, count of objects, spatial
relationship, and visual confusion. We provide a comprehensive analysis of
popular MLLMs, ranging from GPT-4V, Gemini-Pro, to open-sourced models, such as
LLaVA-1.5 and CogVLM. Empirically, we observe significant performance gaps
between GPT-4V and other models; and previous robust instruction-tuned models,
such as LRV-Instruction and LLaVA-RLHF, are not effective on this new
benchmark. While GPT-4V achieves 75.02% accuracy on MAD-Bench, the accuracy of
any other model in our experiments ranges from 5% to 35%. We further propose a
remedy that adds an additional paragraph to the deceptive prompts to encourage
models to think twice before answering the question. Surprisingly, this simple
method can even double the accuracy; however, the absolute numbers are still
too low to be satisfactory. We hope MAD-Bench can serve as a valuable benchmark
to stimulate further research to enhance models' resilience against deceptive
prompts.
MAD-Bench is a benchmark designed to evaluate the susceptibility of Multimodal Large Language Models (MLLMs) to deceptive prompts, with GPT-4V performing significantly better than other models but still facing limitations.
Multimodal Large Language Models (MLLMs) MAD-Bench GPT-4V Gemini-Pro LLaVA-1.5 CogVLM LRV-Instruction LLaVA-RLHF hallucinated responses deceptive prompts