title: TULIP: Towards Unified Language-Image Pretraining
summary: Despite the recent success of image-text contrastive models like CLIP and
  SigLIP, these models often struggle with vision-centric tasks that demand
  high-fidelity image understanding, such as counting, depth estimation, and
  fine-grained object recognition. These models, by performing language
  alignment, tend to prioritize high-level semantics over visual understanding,
  weakening their image understanding. On the other hand, vision-focused models
  are great at processing visual information but struggle to understand language,
  limiting their flexibility for language-driven tasks. In this work, we
  introduce TULIP, an open-source, drop-in replacement for existing CLIP-like
  models. Our method leverages generative data augmentation, enhanced image-image
  and text-text contrastive learning, and image/text reconstruction
  regularization to learn fine-grained visual features while preserving global
  semantic alignment. Our approach, scaling to over 1B parameters, outperforms
  existing state-of-the-art (SOTA) models across multiple benchmarks,
  establishing a new SOTA zero-shot performance on ImageNet-1K, delivering up to
  a 2times enhancement over SigLIP on RxRx1 in linear probing for few-shot
  classification, and improving vision-language models, achieving over 3times
  higher scores than SigLIP on MMVP. Our code/checkpoints are available at
  https://tulip-berkeley.github.io
paper_authors._id: 67db7dd224fe67fe45b21e63
paper_authors.name: Zineng Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-20T11:06:57.434Z
paper_authors._id: 6326e20bf0e99f96e024a164
paper_authors.avatarUrl: /avatars/9be111af274cad965f72ed48c71d6122.svg
paper_authors.fullname: Zineng Tang
paper_authors.type: user
paper_authors.user: ZinengTang
paper_authors._id: 67db7dd224fe67fe45b21e64
paper_authors.name: Long Lian
paper_authors._id: 67db7dd224fe67fe45b21e65
paper_authors.name: Seun Eisape
paper_authors._id: 67db7dd224fe67fe45b21e66
paper_authors.name: XuDong Wang
paper_authors._id: 67db7dd224fe67fe45b21e67
paper_authors.name: Roei Herzig
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-20T13:19:56.648Z
paper_authors._id: 667c5764186b27ef806636d3
paper_authors.avatarUrl: /avatars/5c08f0109bc0e350624112c0aff544f6.svg
paper_authors.fullname: Roei Herzig
paper_authors.type: user
paper_authors.user: roeiherz
paper_authors._id: 67db7dd224fe67fe45b21e68
paper_authors.name: Adam Yala
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-20T10:45:44.806Z
paper_authors._id: 6333a9195a032dcd095dda13
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1664329996201-noauth.jpeg
paper_authors.fullname: Adam Yala
paper_authors.type: user
paper_authors.user: yala
paper_authors._id: 67db7dd224fe67fe45b21e69
paper_authors.name: Alane Suhr
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-20T13:20:09.062Z
paper_authors._id: 6611e6e1188ff298b0dd0b79
paper_authors.avatarUrl: /avatars/3a495283955ec9e06e1829c7eb2cd9a4.svg
paper_authors.fullname: Alane Suhr
paper_authors.type: user
paper_authors.user: alsuhr
paper_authors._id: 67db7dd224fe67fe45b21e6a
paper_authors.name: Trevor Darrell
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-20T13:20:14.935Z
paper_authors._id: 64cbdf02f103036e23d1c7f3
paper_authors.avatarUrl: /avatars/496069463900dea20929b57381182d39.svg
paper_authors.fullname: Trevor Darrell
paper_authors.type: user
paper_authors.user: trevordarrell
paper_authors._id: 67db7dd224fe67fe45b21e6b
paper_authors.name: David M. Chan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-20T13:20:24.455Z
paper_authors._id: 6388f68c43d8b0797a09ff84
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1669920369989-noauth.jpeg
paper_authors.fullname: David Chan
paper_authors.type: user
paper_authors.user: davidchan
paper_title: TULIP: Towards Unified Language-Image Pretraining
paper_summary: Despite the recent success of image-text contrastive models like CLIP and
  SigLIP, these models often struggle with vision-centric tasks that demand
  high-fidelity image understanding, such as counting, depth estimation, and
  fine-grained object recognition. These models, by performing language
  alignment, tend to prioritize high-level semantics over visual understanding,
  weakening their image understanding. On the other hand, vision-focused models
  are great at processing visual information but struggle to understand language,
  limiting their flexibility for language-driven tasks. In this work, we
  introduce TULIP, an open-source, drop-in replacement for existing CLIP-like
  models. Our method leverages generative data augmentation, enhanced image-image
  and text-text contrastive learning, and image/text reconstruction
  regularization to learn fine-grained visual features while preserving global
  semantic alignment. Our approach, scaling to over 1B parameters, outperforms
  existing state-of-the-art (SOTA) models across multiple benchmarks,
  establishing a new SOTA zero-shot performance on ImageNet-1K, delivering up to
  a 2times enhancement over SigLIP on RxRx1 in linear probing for few-shot
  classification, and improving vision-language models, achieving over 3times
  higher scores than SigLIP on MMVP. Our code/checkpoints are available at
  https://tulip-berkeley.github.io
paper_ai_summary: TULIP enhances image-text contrastive models by incorporating generative data augmentation and contrastive learning to improve fine-grained visual features and zero-shot performance.
paper_ai_keywords: CLIP SigLIP vision-centric tasks high-fidelity image understanding counting depth estimation fine-grained object recognition language alignment generative data augmentation image-image contrastive learning text-text contrastive learning image/text reconstruction regularization ImageNet-1K RxRx1 zero-shot performance few-shot classification vision-language models MMVP