Can LLMs Deceive CLIP? Benchmarking Adversarial Compositionality of
  Pre-trained Multimodal Representation via Text Updates
While pre-trained multimodal representations (e.g., CLIP) have shown
impressive capabilities, they exhibit significant compositional vulnerabilities
leading to counterintuitive judgments. We introduce Multimodal Adversarial
Compositionality (MAC), a benchmark that leverages large language models (LLMs)
to generate deceptive text samples to exploit these vulnerabilities across
different modalities and evaluates them through both sample-wise attack success
rate and group-wise entropy-based diversity. To improve zero-shot methods, we
propose a self-training approach that leverages rejection-sampling fine-tuning
with diversity-promoting filtering, which enhances both attack success rate and
sample diversity. Using smaller language models like Llama-3.1-8B, our approach
demonstrates superior performance in revealing compositional vulnerabilities
across various multimodal representations, including images, videos, and
audios.
683912ee831cc04b0a3c0c6c
Jaewoo Ahn
claimed_verified
2025-05-30T06:53:54.580Z
64bb081c01f1983a863654dc
/avatars/038894bc72b92ec3f4ecb096cc60b60a.svg
Jaewoo Ahn
user
ahnpersie
683912ee831cc04b0a3c0c6d
Heeseung Yun
683912ee831cc04b0a3c0c6e
Dayoon Ko
683912ee831cc04b0a3c0c6f
Gunhee Kim
Can LLMs Deceive CLIP? Benchmarking Adversarial Compositionality of
  Pre-trained Multimodal Representation via Text Updates
While pre-trained multimodal representations (e.g., CLIP) have shown
impressive capabilities, they exhibit significant compositional vulnerabilities
leading to counterintuitive judgments. We introduce Multimodal Adversarial
Compositionality (MAC), a benchmark that leverages large language models (LLMs)
to generate deceptive text samples to exploit these vulnerabilities across
different modalities and evaluates them through both sample-wise attack success
rate and group-wise entropy-based diversity. To improve zero-shot methods, we
propose a self-training approach that leverages rejection-sampling fine-tuning
with diversity-promoting filtering, which enhances both attack success rate and
sample diversity. Using smaller language models like Llama-3.1-8B, our approach
demonstrates superior performance in revealing compositional vulnerabilities
across various multimodal representations, including images, videos, and
audios.
A benchmark using deceptive text samples to evaluate compositional vulnerabilities in multimodal representations is introduced, and a self-training approach improves zero-shot methods by enhancing attack success and sample diversity.
Multimodal Adversarial Compositionality (MAC) multimodal representations large language models (LLMs) rejection-sampling fine-tuning diversity-promoting filtering images videos audios