VLSBench: Unveiling Visual Leakage in Multimodal Safety
Safety concerns of Multimodal large language models (MLLMs) have gradually
become an important problem in various applications. Surprisingly, previous
works indicate a counter-intuitive phenomenon that using textual unlearning to
align MLLMs achieves comparable safety performances with MLLMs trained with
image-text pairs. To explain such a counter-intuitive phenomenon, we discover a
visual safety information leakage (VSIL) problem in existing multimodal safety
benchmarks, i.e., the potentially risky and sensitive content in the image has
been revealed in the textual query. In this way, MLLMs can easily refuse these
sensitive text-image queries according to textual queries. However, image-text
pairs without VSIL are common in real-world scenarios and are overlooked by
existing multimodal safety benchmarks. To this end, we construct multimodal
visual leakless safety benchmark (VLSBench) preventing visual safety leakage
from image to textual query with 2.4k image-text pairs. Experimental results
indicate that VLSBench poses a significant challenge to both open-source and
close-source MLLMs, including LLaVA, Qwen2-VL, Llama3.2-Vision, and GPT-4o.
This study demonstrates that textual alignment is enough for multimodal safety
scenarios with VSIL, while multimodal alignment is a more promising solution
for multimodal safety scenarios without VSIL. Please see our code and data at:
http://hxhcreate.github.io/VLSBench
674d36af4f75f81717c47767
Xuhao Hu
extracted_confirmed
2024-12-02T04:27:20.801Z
6372813520a58a5e14c596a3
/avatars/9135151259db3e5b9c8969e1d00c949d.svg
XuHao Hu
user
Foreshhh
674d36af4f75f81717c47768
Dongrui Liu
admin_assigned
2024-12-03T11:25:48.997Z
657fe7a8504da7f6f30a2832
/avatars/65987e3cba449b5d250616510ee11f33.svg
Dongrui Liu
user
Max9803
674d36af4f75f81717c47769
Hao Li
claimed_verified
2024-12-03T11:16:13.054Z
653f1ef4aabbf15fc76a259c
/avatars/94e569999d913e961266394ea2875965.svg
LLLeo Li
user
LLLeo612
674d36af4f75f81717c4776a
Xuanjing Huang
674d36af4f75f81717c4776b
Jing Shao
VLSBench: Unveiling Visual Leakage in Multimodal Safety
Safety concerns of Multimodal large language models (MLLMs) have gradually
become an important problem in various applications. Surprisingly, previous
works indicate a counter-intuitive phenomenon that using textual unlearning to
align MLLMs achieves comparable safety performances with MLLMs trained with
image-text pairs. To explain such a counter-intuitive phenomenon, we discover a
visual safety information leakage (VSIL) problem in existing multimodal safety
benchmarks, i.e., the potentially risky and sensitive content in the image has
been revealed in the textual query. In this way, MLLMs can easily refuse these
sensitive text-image queries according to textual queries. However, image-text
pairs without VSIL are common in real-world scenarios and are overlooked by
existing multimodal safety benchmarks. To this end, we construct multimodal
visual leakless safety benchmark (VLSBench) preventing visual safety leakage
from image to textual query with 2.4k image-text pairs. Experimental results
indicate that VLSBench poses a significant challenge to both open-source and
close-source MLLMs, including LLaVA, Qwen2-VL, Llama3.2-Vision, and GPT-4o.
This study demonstrates that textual alignment is enough for multimodal safety
scenarios with VSIL, while multimodal alignment is a more promising solution
for multimodal safety scenarios without VSIL. Please see our code and data at:
http://hxhcreate.github.io/VLSBench
The study reveals that visual safety information leakage (VSIL) in multimodal benchmarks affects the performance of MLLMs, highlighting the need for multimodal alignment to ensure safety in VSIL-free scenarios.
Multimodal large language models textual unlearning visual safety information leakage VSIL multimodal safety benchmarks multimodal visual leakless safety benchmark VLSBench LLaVA Qwen2-VL Llama3.2-Vision GPT-4o