Are Vision-Language Models Safe in the Wild? A Meme-Based Benchmark
  Study
Rapid deployment of vision-language models (VLMs) magnifies safety risks, yet
most evaluations rely on artificial images. This study asks: How safe are
current VLMs when confronted with meme images that ordinary users share? To
investigate this question, we introduce MemeSafetyBench, a 50,430-instance
benchmark pairing real meme images with both harmful and benign instructions.
Using a comprehensive safety taxonomy and LLM-based instruction generation, we
assess multiple VLMs across single and multi-turn interactions. We investigate
how real-world memes influence harmful outputs, the mitigating effects of
conversational context, and the relationship between model scale and safety
metrics. Our findings demonstrate that VLMs show greater vulnerability to
meme-based harmful prompts than to synthetic or typographic images. Memes
significantly increase harmful responses and decrease refusals compared to
text-only inputs. Though multi-turn interactions provide partial mitigation,
elevated vulnerability persists. These results highlight the need for
ecologically valid evaluations and stronger safety mechanisms.
682f518184a99219c4b3090c
DongGeon Lee
claimed_verified
2025-05-26T08:17:11.519Z
6540fbf9cb7fffd683942b43
/avatars/d4a64fbde511d0949e1c339179586850.svg
DongGeon Lee
user
oneonlee
682f518184a99219c4b3090d
Joonwon Jang
claimed_verified
2025-07-16T15:16:01.384Z
64bf77338e051085ba405d66
/avatars/dd8b1f976c67c8b592f7d1e88573fa8b.svg
Joonwon
user
lainshower
682f518184a99219c4b3090e
Jihae Jeong
claimed_verified
2025-05-28T09:01:18.400Z
65a9fdf39089149c5d0f5305
/avatars/1c134be2cd9aaebd6e4b2cca074ad9b9.svg
JIHAE JEONG
user
jjae
682f518184a99219c4b3090f
Hwanjo Yu
Are Vision-Language Models Safe in the Wild? A Meme-Based Benchmark
  Study
Rapid deployment of vision-language models (VLMs) magnifies safety risks, yet
most evaluations rely on artificial images. This study asks: How safe are
current VLMs when confronted with meme images that ordinary users share? To
investigate this question, we introduce MemeSafetyBench, a 50,430-instance
benchmark pairing real meme images with both harmful and benign instructions.
Using a comprehensive safety taxonomy and LLM-based instruction generation, we
assess multiple VLMs across single and multi-turn interactions. We investigate
how real-world memes influence harmful outputs, the mitigating effects of
conversational context, and the relationship between model scale and safety
metrics. Our findings demonstrate that VLMs show greater vulnerability to
meme-based harmful prompts than to synthetic or typographic images. Memes
significantly increase harmful responses and decrease refusals compared to
text-only inputs. Though multi-turn interactions provide partial mitigation,
elevated vulnerability persists. These results highlight the need for
ecologically valid evaluations and stronger safety mechanisms.
VLMs are more vulnerable to harmful meme-based prompts than to synthetic images, and while multi-turn interactions offer some protection, significant vulnerabilities remain.
vision-language models VLMs MemeSafetyBench safety taxonomy LLM-based instruction generation single and multi-turn interactions