title: Safeguarding Vision-Language Models: Mitigating Vulnerabilities to
    Gaussian Noise in Perturbation-based Attacks
summary: Vision-Language Models (VLMs) extend the capabilities of Large Language
  Models (LLMs) by incorporating visual information, yet they remain vulnerable
  to jailbreak attacks, especially when processing noisy or corrupted images.
  Although existing VLMs adopt security measures during training to mitigate such
  attacks, vulnerabilities associated with noise-augmented visual inputs are
  overlooked. In this work, we identify that missing noise-augmented training
  causes critical security gaps: many VLMs are susceptible to even simple
  perturbations such as Gaussian noise. To address this challenge, we propose
  Robust-VLGuard, a multimodal safety dataset with aligned / misaligned
  image-text pairs, combined with noise-augmented fine-tuning that reduces attack
  success rates while preserving functionality of VLM. For stronger
  optimization-based visual perturbation attacks, we propose DiffPure-VLM,
  leveraging diffusion models to convert adversarial perturbations into
  Gaussian-like noise, which can be defended by VLMs with noise-augmented safety
  fine-tuning. Experimental results demonstrate that the distribution-shifting
  property of diffusion model aligns well with our fine-tuned VLMs, significantly
  mitigating adversarial perturbations across varying intensities. The dataset
  and code are available at https://github.com/JarvisUSTC/DiffPure-RobustVLM.
paper_authors._id: 67ede544ed9c94861b82b29f
paper_authors.name: Jiawei Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-03T07:52:27.815Z
paper_authors._id: 64060b49a577649430bf6974
paper_authors.avatarUrl: /avatars/74d0d6ed656b593e4c101b09edf18c7a.svg
paper_authors.fullname: Jiawei Wang
paper_authors.type: user
paper_authors.user: Jarvis1111
paper_authors._id: 67ede544ed9c94861b82b2a0
paper_authors.name: Yushen Zuo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-03T14:04:22.576Z
paper_authors._id: 643e9efa2263cdc630f88f5c
paper_authors.avatarUrl: /avatars/96cea51f17e7d41ffb6a4b438e05f5cb.svg
paper_authors.fullname: Yushen Zuo
paper_authors.type: user
paper_authors.user: YSZuo
paper_authors._id: 67ede544ed9c94861b82b2a1
paper_authors.name: Yuanjun Chai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-03T07:52:23.962Z
paper_authors._id: 64756323d815855e4ef945a0
paper_authors.avatarUrl: /avatars/29f5150805dafce2b3f9da441c8be988.svg
paper_authors.fullname: Chai
paper_authors.type: user
paper_authors.user: AllenChai
paper_authors._id: 67ede544ed9c94861b82b2a2
paper_authors.name: Zhendong Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-03T14:04:31.772Z
paper_authors._id: 6267a1aff91d1c1633bf3fc0
paper_authors.avatarUrl: /avatars/c3a6be3e4cea8c447ea2ba18059a566c.svg
paper_authors.fullname: ZhenDong Liu
paper_authors.type: user
paper_authors.user: Zaneec
paper_authors._id: 67ede544ed9c94861b82b2a3
paper_authors.name: Yichen Fu
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-04-03T01:32:57.886Z
paper_authors._id: 6528ce81598467feb33d992d
paper_authors.avatarUrl: /avatars/e98a7bf16e6fd5118e861d562f93bb9b.svg
paper_authors.fullname: Yicheng Fu
paper_authors.type: user
paper_authors.user: sofyc
paper_authors._id: 67ede544ed9c94861b82b2a4
paper_authors.name: Yichun Feng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-03T14:04:42.397Z
paper_authors._id: 67b6e2769fd5603fe15b84b9
paper_authors.avatarUrl: /avatars/e6b42c49954a840518bc3e3aa2617912.svg
paper_authors.fullname: fengyichun
paper_authors.type: user
paper_authors.user: fengyichun
paper_authors._id: 67ede544ed9c94861b82b2a5
paper_authors.name: Kin-man Lam
paper_title: Safeguarding Vision-Language Models: Mitigating Vulnerabilities to
    Gaussian Noise in Perturbation-based Attacks
paper_summary: Vision-Language Models (VLMs) extend the capabilities of Large Language
  Models (LLMs) by incorporating visual information, yet they remain vulnerable
  to jailbreak attacks, especially when processing noisy or corrupted images.
  Although existing VLMs adopt security measures during training to mitigate such
  attacks, vulnerabilities associated with noise-augmented visual inputs are
  overlooked. In this work, we identify that missing noise-augmented training
  causes critical security gaps: many VLMs are susceptible to even simple
  perturbations such as Gaussian noise. To address this challenge, we propose
  Robust-VLGuard, a multimodal safety dataset with aligned / misaligned
  image-text pairs, combined with noise-augmented fine-tuning that reduces attack
  success rates while preserving functionality of VLM. For stronger
  optimization-based visual perturbation attacks, we propose DiffPure-VLM,
  leveraging diffusion models to convert adversarial perturbations into
  Gaussian-like noise, which can be defended by VLMs with noise-augmented safety
  fine-tuning. Experimental results demonstrate that the distribution-shifting
  property of diffusion model aligns well with our fine-tuned VLMs, significantly
  mitigating adversarial perturbations across varying intensities. The dataset
  and code are available at https://github.com/JarvisUSTC/DiffPure-RobustVLM.
paper_ai_summary: Robust-VLGuard and DiffPure-VLM enhance Vision-Language Models by mitigating vulnerabilities to adversarial perturbations through noise-augmented training and diffusion models.
paper_ai_keywords: Vision-Language Models Large Language Models noise-augmented training adversarial perturbations Gaussian noise multimodal safety dataset noise-augmented fine-tuning diffusion models distribution-shifting property DiffPure-VLM Robust-VLGuard