title: DiffCLIP: Differential Attention Meets CLIP
summary: We propose DiffCLIP, a novel vision-language model that extends the
  differential attention mechanism to CLIP architectures. Differential attention
  was originally developed for large language models to amplify relevant context
  while canceling out noisy information. In this work, we integrate this
  mechanism into CLIP's dual encoder (image and text) framework. With minimal
  additional parameters, DiffCLIP achieves superior performance on image-text
  understanding tasks. Across zero-shot classification, retrieval, and robustness
  benchmarks, DiffCLIP consistently outperforms baseline CLIP models. Notably,
  these gains come with negligible computational overhead, demonstrating that
  differential attention can significantly enhance multi-modal representations
  without sacrificing efficiency. Code can be found at
  https://github.com/hammoudhasan/DiffCLIP.
paper_authors._id: 67d02a6da2b648cac0c37359
paper_authors.name: Hasan Abed Al Kader Hammoud
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T14:30:32.989Z
paper_authors._id: 642b51385bf2355d02a23d15
paper_authors.avatarUrl: /avatars/87985347643b2647555f2453fa4d94fb.svg
paper_authors.fullname: Hasan Abed Al Kader Hammoud
paper_authors.type: user
paper_authors.user: hammh0a
paper_authors._id: 67d02a6da2b648cac0c3735a
paper_authors.name: Bernard Ghanem
paper_title: DiffCLIP: Differential Attention Meets CLIP
paper_summary: We propose DiffCLIP, a novel vision-language model that extends the
  differential attention mechanism to CLIP architectures. Differential attention
  was originally developed for large language models to amplify relevant context
  while canceling out noisy information. In this work, we integrate this
  mechanism into CLIP's dual encoder (image and text) framework. With minimal
  additional parameters, DiffCLIP achieves superior performance on image-text
  understanding tasks. Across zero-shot classification, retrieval, and robustness
  benchmarks, DiffCLIP consistently outperforms baseline CLIP models. Notably,
  these gains come with negligible computational overhead, demonstrating that
  differential attention can significantly enhance multi-modal representations
  without sacrificing efficiency. Code can be found at
  https://github.com/hammoudhasan/DiffCLIP.
paper_ai_keywords: DiffCLIP differential attention mechanism CLIP architectures dual encoder image-text understanding tasks zero-shot classification retrieval robustness benchmarks