title: VLMGuard: Defending VLMs against Malicious Prompts via Unlabeled Data
summary: Vision-language models (VLMs) are essential for contextual understanding of
  both visual and textual information. However, their vulnerability to
  adversarially manipulated inputs presents significant risks, leading to
  compromised outputs and raising concerns about the reliability in
  VLM-integrated applications. Detecting these malicious prompts is thus crucial
  for maintaining trust in VLM generations. A major challenge in developing a
  safeguarding prompt classifier is the lack of a large amount of labeled benign
  and malicious data. To address the issue, we introduce VLMGuard, a novel
  learning framework that leverages the unlabeled user prompts in the wild for
  malicious prompt detection. These unlabeled prompts, which naturally arise when
  VLMs are deployed in the open world, consist of both benign and malicious
  information. To harness the unlabeled data, we present an automated
  maliciousness estimation score for distinguishing between benign and malicious
  samples within this unlabeled mixture, thereby enabling the training of a
  binary prompt classifier on top. Notably, our framework does not require extra
  human annotations, offering strong flexibility and practicality for real-world
  applications. Extensive experiment shows VLMGuard achieves superior detection
  results, significantly outperforming state-of-the-art methods. Disclaimer: This
  paper may contain offensive examples; reader discretion is advised.
paper_authors._id: 66ff32007d722f0879eceff3
paper_authors.name: Xuefeng Du
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-07T07:28:27.354Z
paper_authors._id: 63374d7a0267ebcf0266c83d
paper_authors.avatarUrl: /avatars/dab490fc1950f2778f5a6e9bf9893aaa.svg
paper_authors.fullname: Xuefeng Du
paper_authors.type: user
paper_authors.user: xfdu1
paper_authors._id: 66ff32007d722f0879eceff4
paper_authors.name: Reshmi Ghosh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-10T11:16:58.659Z
paper_authors._id: 63228cad95d6f717a8c8e1ec
paper_authors.avatarUrl: /avatars/5fa606f63f841879af1a7617366d18b4.svg
paper_authors.fullname: Reshmi Ghosh
paper_authors.type: user
paper_authors.user: reshmighosh
paper_authors._id: 66ff32007d722f0879eceff5
paper_authors.name: Robert Sim
paper_authors._id: 66ff32007d722f0879eceff6
paper_authors.name: Ahmed Salem
paper_authors._id: 66ff32007d722f0879eceff7
paper_authors.name: Vitor Carvalho
paper_authors._id: 66ff32007d722f0879eceff8
paper_authors.name: Emily Lawton
paper_authors._id: 66ff32007d722f0879eceff9
paper_authors.name: Yixuan Li
paper_authors._id: 66ff32007d722f0879eceffa
paper_authors.name: Jack W. Stokes
paper_title: VLMGuard: Defending VLMs against Malicious Prompts via Unlabeled Data
paper_summary: Vision-language models (VLMs) are essential for contextual understanding of
  both visual and textual information. However, their vulnerability to
  adversarially manipulated inputs presents significant risks, leading to
  compromised outputs and raising concerns about the reliability in
  VLM-integrated applications. Detecting these malicious prompts is thus crucial
  for maintaining trust in VLM generations. A major challenge in developing a
  safeguarding prompt classifier is the lack of a large amount of labeled benign
  and malicious data. To address the issue, we introduce VLMGuard, a novel
  learning framework that leverages the unlabeled user prompts in the wild for
  malicious prompt detection. These unlabeled prompts, which naturally arise when
  VLMs are deployed in the open world, consist of both benign and malicious
  information. To harness the unlabeled data, we present an automated
  maliciousness estimation score for distinguishing between benign and malicious
  samples within this unlabeled mixture, thereby enabling the training of a
  binary prompt classifier on top. Notably, our framework does not require extra
  human annotations, offering strong flexibility and practicality for real-world
  applications. Extensive experiment shows VLMGuard achieves superior detection
  results, significantly outperforming state-of-the-art methods. Disclaimer: This
  paper may contain offensive examples; reader discretion is advised.
paper_ai_summary: VLMGuard detects malicious prompts without human labeling by leveraging an automated maliciousness estimation score on unlabeled prompts, achieving superior detection over state-of-the-art methods.
paper_ai_keywords: Vision-language models VLMGuard malicious prompt detection unlabeled data automatic maliciousness estimation binary prompt classifier