title: Can LLMs Deceive CLIP? Benchmarking Adversarial Compositionality of
    Pre-trained Multimodal Representation via Text Updates
summary: While pre-trained multimodal representations (e.g., CLIP) have shown
  impressive capabilities, they exhibit significant compositional vulnerabilities
  leading to counterintuitive judgments. We introduce Multimodal Adversarial
  Compositionality (MAC), a benchmark that leverages large language models (LLMs)
  to generate deceptive text samples to exploit these vulnerabilities across
  different modalities and evaluates them through both sample-wise attack success
  rate and group-wise entropy-based diversity. To improve zero-shot methods, we
  propose a self-training approach that leverages rejection-sampling fine-tuning
  with diversity-promoting filtering, which enhances both attack success rate and
  sample diversity. Using smaller language models like Llama-3.1-8B, our approach
  demonstrates superior performance in revealing compositional vulnerabilities
  across various multimodal representations, including images, videos, and
  audios.
paper_authors._id: 683912ee831cc04b0a3c0c6c
paper_authors.name: Jaewoo Ahn
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-30T06:53:54.580Z
paper_authors._id: 64bb081c01f1983a863654dc
paper_authors.avatarUrl: /avatars/038894bc72b92ec3f4ecb096cc60b60a.svg
paper_authors.fullname: Jaewoo Ahn
paper_authors.type: user
paper_authors.user: ahnpersie
paper_authors._id: 683912ee831cc04b0a3c0c6d
paper_authors.name: Heeseung Yun
paper_authors._id: 683912ee831cc04b0a3c0c6e
paper_authors.name: Dayoon Ko
paper_authors._id: 683912ee831cc04b0a3c0c6f
paper_authors.name: Gunhee Kim
paper_title: Can LLMs Deceive CLIP? Benchmarking Adversarial Compositionality of
    Pre-trained Multimodal Representation via Text Updates
paper_summary: While pre-trained multimodal representations (e.g., CLIP) have shown
  impressive capabilities, they exhibit significant compositional vulnerabilities
  leading to counterintuitive judgments. We introduce Multimodal Adversarial
  Compositionality (MAC), a benchmark that leverages large language models (LLMs)
  to generate deceptive text samples to exploit these vulnerabilities across
  different modalities and evaluates them through both sample-wise attack success
  rate and group-wise entropy-based diversity. To improve zero-shot methods, we
  propose a self-training approach that leverages rejection-sampling fine-tuning
  with diversity-promoting filtering, which enhances both attack success rate and
  sample diversity. Using smaller language models like Llama-3.1-8B, our approach
  demonstrates superior performance in revealing compositional vulnerabilities
  across various multimodal representations, including images, videos, and
  audios.
paper_ai_summary: A benchmark using deceptive text samples to evaluate compositional vulnerabilities in multimodal representations is introduced, and a self-training approach improves zero-shot methods by enhancing attack success and sample diversity.
paper_ai_keywords: Multimodal Adversarial Compositionality (MAC) multimodal representations large language models (LLMs) rejection-sampling fine-tuning diversity-promoting filtering images videos audios