title: Robustness in Both Domains: CLIP Needs a Robust Text Encoder
summary: Adversarial input attacks can cause a significant shift of CLIP embeddings.
  This can affect the downstream robustness of models incorporating CLIP in the
  pipeline, such as text-to-image generative models or large vision language
  models. While some efforts have been done towards making the CLIP image
  encoders robust, the robustness of text encoders remains unexplored. In this
  work, we cover this gap in the literature. We propose LEAF: an efficient
  adversarial finetuning method for the text domain, with the ability to scale to
  large CLIP models. Our models significantly improve the zero-shot adversarial
  accuracy in the text domain, while maintaining the vision performance provided
  by robust image encoders. When combined with text-to-image diffusion models, we
  can improve the generation quality under adversarial noise. When employing our
  robust CLIP encoders in multimodal retrieval tasks, we improve the recall under
  adversarial noise over standard CLIP models. Finally, we show that robust text
  encoders facilitate better reconstruction of input text from its embedding via
  direct optimization.
paper_authors._id: 68415c5cce09e3eca94e9839
paper_authors.name: Elias Abad Rocamora
paper_authors._id: 68415c5cce09e3eca94e983a
paper_authors.name: Christian Schlarmann
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-05T09:59:48.051Z
paper_authors._id: 6310a6bb0a43f97f6c5567d3
paper_authors.avatarUrl: /avatars/04b07c3a6c811337939c951567cb2bf2.svg
paper_authors.fullname: Christian Schlarmann
paper_authors.type: user
paper_authors.user: chs20
paper_authors._id: 68415c5cce09e3eca94e983b
paper_authors.name: Naman Deep Singh
paper_authors._id: 68415c5cce09e3eca94e983c
paper_authors.name: Yongtao Wu
paper_authors._id: 68415c5cce09e3eca94e983d
paper_authors.name: Matthias Hein
paper_authors._id: 68415c5cce09e3eca94e983e
paper_authors.name: Volkan Cevher
paper_title: Robustness in Both Domains: CLIP Needs a Robust Text Encoder
paper_summary: Adversarial input attacks can cause a significant shift of CLIP embeddings.
  This can affect the downstream robustness of models incorporating CLIP in the
  pipeline, such as text-to-image generative models or large vision language
  models. While some efforts have been done towards making the CLIP image
  encoders robust, the robustness of text encoders remains unexplored. In this
  work, we cover this gap in the literature. We propose LEAF: an efficient
  adversarial finetuning method for the text domain, with the ability to scale to
  large CLIP models. Our models significantly improve the zero-shot adversarial
  accuracy in the text domain, while maintaining the vision performance provided
  by robust image encoders. When combined with text-to-image diffusion models, we
  can improve the generation quality under adversarial noise. When employing our
  robust CLIP encoders in multimodal retrieval tasks, we improve the recall under
  adversarial noise over standard CLIP models. Finally, we show that robust text
  encoders facilitate better reconstruction of input text from its embedding via
  direct optimization.
paper_ai_summary: LEAF, an adversarial finetuning method, enhances the robustness of CLIP text encoders, improving zero-shot accuracy and multimodal retrieval performance under adversarial noise.
paper_ai_keywords: adversarial input attacks CLIP embeddings text-to-image generative models large vision language models adversarial finetuning zero-shot adversarial accuracy text-to-image diffusion models multimodal retrieval tasks robust text encoders