title: How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on
    Deceptive Prompts
summary: The remarkable advancements in Multimodal Large Language Models (MLLMs) have
  not rendered them immune to challenges, particularly in the context of handling
  deceptive information in prompts, thus producing hallucinated responses under
  such conditions. To quantitatively assess this vulnerability, we present
  MAD-Bench, a carefully curated benchmark that contains 850 test samples divided
  into 6 categories, such as non-existent objects, count of objects, spatial
  relationship, and visual confusion. We provide a comprehensive analysis of
  popular MLLMs, ranging from GPT-4V, Gemini-Pro, to open-sourced models, such as
  LLaVA-1.5 and CogVLM. Empirically, we observe significant performance gaps
  between GPT-4V and other models; and previous robust instruction-tuned models,
  such as LRV-Instruction and LLaVA-RLHF, are not effective on this new
  benchmark. While GPT-4V achieves 75.02% accuracy on MAD-Bench, the accuracy of
  any other model in our experiments ranges from 5% to 35%. We further propose a
  remedy that adds an additional paragraph to the deceptive prompts to encourage
  models to think twice before answering the question. Surprisingly, this simple
  method can even double the accuracy; however, the absolute numbers are still
  too low to be satisfactory. We hope MAD-Bench can serve as a valuable benchmark
  to stimulate further research to enhance models' resilience against deceptive
  prompts.
paper_authors._id: 65d56bb3c81e8c3cffd3d0c0
paper_authors.name: Yusu Qian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-21T11:19:43.043Z
paper_authors._id: 65d5d6e590119106db895670
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65d5d6e590119106db895670/SphTxh2E0MtT8T_bH0_zb.png
paper_authors.fullname: Yusu Qian
paper_authors.type: user
paper_authors.user: YusuQian
paper_authors._id: 65d56bb3c81e8c3cffd3d0c1
paper_authors.name: Haotian Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-15T08:05:58.728Z
paper_authors._id: 631516348d85ad332fa47b2c
paper_authors.avatarUrl: /avatars/100f5ae3cf3c52faaecdaecd5d8f2881.svg
paper_authors.fullname: Haotian Zhang
paper_authors.type: user
paper_authors.user: haotiz
paper_authors._id: 65d56bb3c81e8c3cffd3d0c2
paper_authors.name: Yinfei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-21T10:48:58.193Z
paper_authors._id: 64b762568c632fbca942a405
paper_authors.avatarUrl: /avatars/1eb737ec169967872f1ebf5ff29f1e6b.svg
paper_authors.fullname: Yinfei Yang
paper_authors.type: user
paper_authors.user: yinfeiy
paper_authors._id: 65d56bb3c81e8c3cffd3d0c3
paper_authors.name: Zhe Gan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-21T10:49:14.517Z
paper_authors._id: 644022db21ec7c165d848b6e
paper_authors.avatarUrl: /avatars/d9ed37ec77f2682878b2976a04d2fcd8.svg
paper_authors.fullname: Zhe Gan
paper_authors.type: user
paper_authors.user: zhegan27
paper_title: How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on
    Deceptive Prompts
paper_summary: The remarkable advancements in Multimodal Large Language Models (MLLMs) have
  not rendered them immune to challenges, particularly in the context of handling
  deceptive information in prompts, thus producing hallucinated responses under
  such conditions. To quantitatively assess this vulnerability, we present
  MAD-Bench, a carefully curated benchmark that contains 850 test samples divided
  into 6 categories, such as non-existent objects, count of objects, spatial
  relationship, and visual confusion. We provide a comprehensive analysis of
  popular MLLMs, ranging from GPT-4V, Gemini-Pro, to open-sourced models, such as
  LLaVA-1.5 and CogVLM. Empirically, we observe significant performance gaps
  between GPT-4V and other models; and previous robust instruction-tuned models,
  such as LRV-Instruction and LLaVA-RLHF, are not effective on this new
  benchmark. While GPT-4V achieves 75.02% accuracy on MAD-Bench, the accuracy of
  any other model in our experiments ranges from 5% to 35%. We further propose a
  remedy that adds an additional paragraph to the deceptive prompts to encourage
  models to think twice before answering the question. Surprisingly, this simple
  method can even double the accuracy; however, the absolute numbers are still
  too low to be satisfactory. We hope MAD-Bench can serve as a valuable benchmark
  to stimulate further research to enhance models' resilience against deceptive
  prompts.
paper_ai_summary: MAD-Bench is a benchmark designed to evaluate the susceptibility of Multimodal Large Language Models (MLLMs) to deceptive prompts, with GPT-4V performing significantly better than other models but still facing limitations.
paper_ai_keywords: Multimodal Large Language Models (MLLMs) MAD-Bench GPT-4V Gemini-Pro LLaVA-1.5 CogVLM LRV-Instruction LLaVA-RLHF hallucinated responses deceptive prompts