Robustness in Both Domains: CLIP Needs a Robust Text Encoder
Adversarial input attacks can cause a significant shift of CLIP embeddings.
This can affect the downstream robustness of models incorporating CLIP in the
pipeline, such as text-to-image generative models or large vision language
models. While some efforts have been done towards making the CLIP image
encoders robust, the robustness of text encoders remains unexplored. In this
work, we cover this gap in the literature. We propose LEAF: an efficient
adversarial finetuning method for the text domain, with the ability to scale to
large CLIP models. Our models significantly improve the zero-shot adversarial
accuracy in the text domain, while maintaining the vision performance provided
by robust image encoders. When combined with text-to-image diffusion models, we
can improve the generation quality under adversarial noise. When employing our
robust CLIP encoders in multimodal retrieval tasks, we improve the recall under
adversarial noise over standard CLIP models. Finally, we show that robust text
encoders facilitate better reconstruction of input text from its embedding via
direct optimization.
68415c5cce09e3eca94e9839
Elias Abad Rocamora
claimed_verified
2025-06-10T09:22:38.229Z
64add55e7ac35b7de073ff11
https://cdn-avatars.huggingface.co/v1/production/uploads/64add55e7ac35b7de073ff11/82898AE_jYdWMbj5jhC4s.jpeg
Elias Abad Rocamora
user
megaelius
68415c5cce09e3eca94e983a
Christian Schlarmann
claimed_verified
2025-06-05T09:59:48.051Z
6310a6bb0a43f97f6c5567d3
/avatars/04b07c3a6c811337939c951567cb2bf2.svg
Christian Schlarmann
user
chs20
68415c5cce09e3eca94e983b
Naman Deep Singh
68415c5cce09e3eca94e983c
Yongtao Wu
68415c5cce09e3eca94e983d
Matthias Hein
68415c5cce09e3eca94e983e
Volkan Cevher
Robustness in Both Domains: CLIP Needs a Robust Text Encoder
Adversarial input attacks can cause a significant shift of CLIP embeddings.
This can affect the downstream robustness of models incorporating CLIP in the
pipeline, such as text-to-image generative models or large vision language
models. While some efforts have been done towards making the CLIP image
encoders robust, the robustness of text encoders remains unexplored. In this
work, we cover this gap in the literature. We propose LEAF: an efficient
adversarial finetuning method for the text domain, with the ability to scale to
large CLIP models. Our models significantly improve the zero-shot adversarial
accuracy in the text domain, while maintaining the vision performance provided
by robust image encoders. When combined with text-to-image diffusion models, we
can improve the generation quality under adversarial noise. When employing our
robust CLIP encoders in multimodal retrieval tasks, we improve the recall under
adversarial noise over standard CLIP models. Finally, we show that robust text
encoders facilitate better reconstruction of input text from its embedding via
direct optimization.
LEAF, an adversarial finetuning method, enhances the robustness of CLIP text encoders, improving zero-shot accuracy and multimodal retrieval performance under adversarial noise.
adversarial input attacks CLIP embeddings text-to-image generative models large vision language models adversarial finetuning zero-shot adversarial accuracy text-to-image diffusion models multimodal retrieval tasks robust text encoders