title: VLSBench: Unveiling Visual Leakage in Multimodal Safety
summary: Safety concerns of Multimodal large language models (MLLMs) have gradually
  become an important problem in various applications. Surprisingly, previous
  works indicate a counter-intuitive phenomenon that using textual unlearning to
  align MLLMs achieves comparable safety performances with MLLMs trained with
  image-text pairs. To explain such a counter-intuitive phenomenon, we discover a
  visual safety information leakage (VSIL) problem in existing multimodal safety
  benchmarks, i.e., the potentially risky and sensitive content in the image has
  been revealed in the textual query. In this way, MLLMs can easily refuse these
  sensitive text-image queries according to textual queries. However, image-text
  pairs without VSIL are common in real-world scenarios and are overlooked by
  existing multimodal safety benchmarks. To this end, we construct multimodal
  visual leakless safety benchmark (VLSBench) preventing visual safety leakage
  from image to textual query with 2.4k image-text pairs. Experimental results
  indicate that VLSBench poses a significant challenge to both open-source and
  close-source MLLMs, including LLaVA, Qwen2-VL, Llama3.2-Vision, and GPT-4o.
  This study demonstrates that textual alignment is enough for multimodal safety
  scenarios with VSIL, while multimodal alignment is a more promising solution
  for multimodal safety scenarios without VSIL. Please see our code and data at:
  http://hxhcreate.github.io/VLSBench
paper_authors._id: 674d36af4f75f81717c47767
paper_authors.name: Xuhao Hu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-12-02T04:27:20.801Z
paper_authors._id: 6372813520a58a5e14c596a3
paper_authors.avatarUrl: /avatars/9135151259db3e5b9c8969e1d00c949d.svg
paper_authors.fullname: XuHao Hu
paper_authors.type: user
paper_authors.user: Foreshhh
paper_authors._id: 674d36af4f75f81717c47768
paper_authors.name: Dongrui Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T11:25:48.997Z
paper_authors._id: 657fe7a8504da7f6f30a2832
paper_authors.avatarUrl: /avatars/65987e3cba449b5d250616510ee11f33.svg
paper_authors.fullname: Dongrui Liu
paper_authors.type: user
paper_authors.user: Max9803
paper_authors._id: 674d36af4f75f81717c47769
paper_authors.name: Hao Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-03T11:16:13.054Z
paper_authors._id: 653f1ef4aabbf15fc76a259c
paper_authors.avatarUrl: /avatars/94e569999d913e961266394ea2875965.svg
paper_authors.fullname: LLLeo Li
paper_authors.type: user
paper_authors.user: LLLeo612
paper_authors._id: 674d36af4f75f81717c4776a
paper_authors.name: Xuanjing Huang
paper_authors._id: 674d36af4f75f81717c4776b
paper_authors.name: Jing Shao
paper_title: VLSBench: Unveiling Visual Leakage in Multimodal Safety
paper_summary: Safety concerns of Multimodal large language models (MLLMs) have gradually
  become an important problem in various applications. Surprisingly, previous
  works indicate a counter-intuitive phenomenon that using textual unlearning to
  align MLLMs achieves comparable safety performances with MLLMs trained with
  image-text pairs. To explain such a counter-intuitive phenomenon, we discover a
  visual safety information leakage (VSIL) problem in existing multimodal safety
  benchmarks, i.e., the potentially risky and sensitive content in the image has
  been revealed in the textual query. In this way, MLLMs can easily refuse these
  sensitive text-image queries according to textual queries. However, image-text
  pairs without VSIL are common in real-world scenarios and are overlooked by
  existing multimodal safety benchmarks. To this end, we construct multimodal
  visual leakless safety benchmark (VLSBench) preventing visual safety leakage
  from image to textual query with 2.4k image-text pairs. Experimental results
  indicate that VLSBench poses a significant challenge to both open-source and
  close-source MLLMs, including LLaVA, Qwen2-VL, Llama3.2-Vision, and GPT-4o.
  This study demonstrates that textual alignment is enough for multimodal safety
  scenarios with VSIL, while multimodal alignment is a more promising solution
  for multimodal safety scenarios without VSIL. Please see our code and data at:
  http://hxhcreate.github.io/VLSBench
paper_ai_keywords: Multimodal large language models (MLLMs) Textual unlearning Safety performances Visual safety information leakage (VSIL) Multimodal safety benchmarks Sensitive text-image queries Multimodal visual leakless safety benchmark (VLSBench) Image-text pairs Open-source MLLMs Close-source MLLMs LLaVA": "", "Qwen2-VL": "", "Llama3.2-Vision": "", "GPT-4o": ""} Textual alignment Multimodal alignment