VLMGuard: Defending VLMs against Malicious Prompts via Unlabeled Data
Vision-language models (VLMs) are essential for contextual understanding of
both visual and textual information. However, their vulnerability to
adversarially manipulated inputs presents significant risks, leading to
compromised outputs and raising concerns about the reliability in
VLM-integrated applications. Detecting these malicious prompts is thus crucial
for maintaining trust in VLM generations. A major challenge in developing a
safeguarding prompt classifier is the lack of a large amount of labeled benign
and malicious data. To address the issue, we introduce VLMGuard, a novel
learning framework that leverages the unlabeled user prompts in the wild for
malicious prompt detection. These unlabeled prompts, which naturally arise when
VLMs are deployed in the open world, consist of both benign and malicious
information. To harness the unlabeled data, we present an automated
maliciousness estimation score for distinguishing between benign and malicious
samples within this unlabeled mixture, thereby enabling the training of a
binary prompt classifier on top. Notably, our framework does not require extra
human annotations, offering strong flexibility and practicality for real-world
applications. Extensive experiment shows VLMGuard achieves superior detection
results, significantly outperforming state-of-the-art methods. Disclaimer: This
paper may contain offensive examples; reader discretion is advised.
66ff32007d722f0879eceff3
Xuefeng Du
claimed_verified
2024-10-07T07:28:27.354Z
63374d7a0267ebcf0266c83d
/avatars/dab490fc1950f2778f5a6e9bf9893aaa.svg
Xuefeng Du
user
xfdu1
66ff32007d722f0879eceff4
Reshmi Ghosh
claimed_verified
2024-12-10T11:16:58.659Z
63228cad95d6f717a8c8e1ec
/avatars/5fa606f63f841879af1a7617366d18b4.svg
Reshmi Ghosh
user
reshmighosh
66ff32007d722f0879eceff5
Robert Sim
66ff32007d722f0879eceff6
Ahmed Salem
66ff32007d722f0879eceff7
Vitor Carvalho
66ff32007d722f0879eceff8
Emily Lawton
66ff32007d722f0879eceff9
Yixuan Li
66ff32007d722f0879eceffa
Jack W. Stokes
VLMGuard: Defending VLMs against Malicious Prompts via Unlabeled Data
Vision-language models (VLMs) are essential for contextual understanding of
both visual and textual information. However, their vulnerability to
adversarially manipulated inputs presents significant risks, leading to
compromised outputs and raising concerns about the reliability in
VLM-integrated applications. Detecting these malicious prompts is thus crucial
for maintaining trust in VLM generations. A major challenge in developing a
safeguarding prompt classifier is the lack of a large amount of labeled benign
and malicious data. To address the issue, we introduce VLMGuard, a novel
learning framework that leverages the unlabeled user prompts in the wild for
malicious prompt detection. These unlabeled prompts, which naturally arise when
VLMs are deployed in the open world, consist of both benign and malicious
information. To harness the unlabeled data, we present an automated
maliciousness estimation score for distinguishing between benign and malicious
samples within this unlabeled mixture, thereby enabling the training of a
binary prompt classifier on top. Notably, our framework does not require extra
human annotations, offering strong flexibility and practicality for real-world
applications. Extensive experiment shows VLMGuard achieves superior detection
results, significantly outperforming state-of-the-art methods. Disclaimer: This
paper may contain offensive examples; reader discretion is advised.
VLMGuard detects malicious prompts without human labeling by leveraging an automated maliciousness estimation score on unlabeled prompts, achieving superior detection over state-of-the-art methods.
Vision-language models VLMGuard malicious prompt detection unlabeled data automatic maliciousness estimation binary prompt classifier