title: Are Vision-Language Models Safe in the Wild? A Meme-Based Benchmark
    Study
summary: Rapid deployment of vision-language models (VLMs) magnifies safety risks, yet
  most evaluations rely on artificial images. This study asks: How safe are
  current VLMs when confronted with meme images that ordinary users share? To
  investigate this question, we introduce MemeSafetyBench, a 50,430-instance
  benchmark pairing real meme images with both harmful and benign instructions.
  Using a comprehensive safety taxonomy and LLM-based instruction generation, we
  assess multiple VLMs across single and multi-turn interactions. We investigate
  how real-world memes influence harmful outputs, the mitigating effects of
  conversational context, and the relationship between model scale and safety
  metrics. Our findings demonstrate that VLMs show greater vulnerability to
  meme-based harmful prompts than to synthetic or typographic images. Memes
  significantly increase harmful responses and decrease refusals compared to
  text-only inputs. Though multi-turn interactions provide partial mitigation,
  elevated vulnerability persists. These results highlight the need for
  ecologically valid evaluations and stronger safety mechanisms.
paper_authors._id: 682f518184a99219c4b3090c
paper_authors.name: DongGeon Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:17:11.519Z
paper_authors._id: 6540fbf9cb7fffd683942b43
paper_authors.avatarUrl: /avatars/d4a64fbde511d0949e1c339179586850.svg
paper_authors.fullname: DongGeon Lee
paper_authors.type: user
paper_authors.user: oneonlee
paper_authors._id: 682f518184a99219c4b3090d
paper_authors.name: Joonwon Jang
paper_authors._id: 682f518184a99219c4b3090e
paper_authors.name: Jihae Jeong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T09:01:18.400Z
paper_authors._id: 65a9fdf39089149c5d0f5305
paper_authors.avatarUrl: /avatars/1c134be2cd9aaebd6e4b2cca074ad9b9.svg
paper_authors.fullname: JIHAE JEONG
paper_authors.type: user
paper_authors.user: jjae
paper_authors._id: 682f518184a99219c4b3090f
paper_authors.name: Hwanjo Yu
paper_title: Are Vision-Language Models Safe in the Wild? A Meme-Based Benchmark
    Study
paper_summary: Rapid deployment of vision-language models (VLMs) magnifies safety risks, yet
  most evaluations rely on artificial images. This study asks: How safe are
  current VLMs when confronted with meme images that ordinary users share? To
  investigate this question, we introduce MemeSafetyBench, a 50,430-instance
  benchmark pairing real meme images with both harmful and benign instructions.
  Using a comprehensive safety taxonomy and LLM-based instruction generation, we
  assess multiple VLMs across single and multi-turn interactions. We investigate
  how real-world memes influence harmful outputs, the mitigating effects of
  conversational context, and the relationship between model scale and safety
  metrics. Our findings demonstrate that VLMs show greater vulnerability to
  meme-based harmful prompts than to synthetic or typographic images. Memes
  significantly increase harmful responses and decrease refusals compared to
  text-only inputs. Though multi-turn interactions provide partial mitigation,
  elevated vulnerability persists. These results highlight the need for
  ecologically valid evaluations and stronger safety mechanisms.
paper_ai_summary: VLMs are more vulnerable to harmful meme-based prompts than to synthetic images, and while multi-turn interactions offer some protection, significant vulnerabilities remain.
paper_ai_keywords: vision-language models VLMs MemeSafetyBench safety taxonomy LLM-based instruction generation single and multi-turn interactions