TULIP: Towards Unified Language-Image Pretraining
Despite the recent success of image-text contrastive models like CLIP and
SigLIP, these models often struggle with vision-centric tasks that demand
high-fidelity image understanding, such as counting, depth estimation, and
fine-grained object recognition. These models, by performing language
alignment, tend to prioritize high-level semantics over visual understanding,
weakening their image understanding. On the other hand, vision-focused models
are great at processing visual information but struggle to understand language,
limiting their flexibility for language-driven tasks. In this work, we
introduce TULIP, an open-source, drop-in replacement for existing CLIP-like
models. Our method leverages generative data augmentation, enhanced image-image
and text-text contrastive learning, and image/text reconstruction
regularization to learn fine-grained visual features while preserving global
semantic alignment. Our approach, scaling to over 1B parameters, outperforms
existing state-of-the-art (SOTA) models across multiple benchmarks,
establishing a new SOTA zero-shot performance on ImageNet-1K, delivering up to
a 2times enhancement over SigLIP on RxRx1 in linear probing for few-shot
classification, and improving vision-language models, achieving over 3times
higher scores than SigLIP on MMVP. Our code/checkpoints are available at
https://tulip-berkeley.github.io
67db7dd224fe67fe45b21e63
Zineng Tang
admin_assigned
2025-03-20T11:06:57.434Z
6326e20bf0e99f96e024a164
/avatars/9be111af274cad965f72ed48c71d6122.svg
Zineng Tang
user
ZinengTang
67db7dd224fe67fe45b21e64
Long Lian
67db7dd224fe67fe45b21e65
Seun Eisape
67db7dd224fe67fe45b21e66
XuDong Wang
67db7dd224fe67fe45b21e67
Roei Herzig
admin_assigned
2025-03-20T13:19:56.648Z
667c5764186b27ef806636d3
/avatars/5c08f0109bc0e350624112c0aff544f6.svg
Roei Herzig
user
roeiherz
67db7dd224fe67fe45b21e68
Adam Yala
claimed_verified
2025-03-20T10:45:44.806Z
6333a9195a032dcd095dda13
https://cdn-avatars.huggingface.co/v1/production/uploads/1664329996201-noauth.jpeg
Adam Yala
user
yala
67db7dd224fe67fe45b21e69
Alane Suhr
admin_assigned
2025-03-20T13:20:09.062Z
6611e6e1188ff298b0dd0b79
/avatars/3a495283955ec9e06e1829c7eb2cd9a4.svg
Alane Suhr
user
alsuhr
67db7dd224fe67fe45b21e6a
Trevor Darrell
admin_assigned
2025-03-20T13:20:14.935Z
64cbdf02f103036e23d1c7f3
/avatars/496069463900dea20929b57381182d39.svg
Trevor Darrell
user
trevordarrell
67db7dd224fe67fe45b21e6b
David M. Chan
admin_assigned
2025-03-20T13:20:24.455Z
6388f68c43d8b0797a09ff84
https://cdn-avatars.huggingface.co/v1/production/uploads/1669920369989-noauth.jpeg
David Chan
user
davidchan
TULIP: Towards Unified Language-Image Pretraining
Despite the recent success of image-text contrastive models like CLIP and
SigLIP, these models often struggle with vision-centric tasks that demand
high-fidelity image understanding, such as counting, depth estimation, and
fine-grained object recognition. These models, by performing language
alignment, tend to prioritize high-level semantics over visual understanding,
weakening their image understanding. On the other hand, vision-focused models
are great at processing visual information but struggle to understand language,
limiting their flexibility for language-driven tasks. In this work, we
introduce TULIP, an open-source, drop-in replacement for existing CLIP-like
models. Our method leverages generative data augmentation, enhanced image-image
and text-text contrastive learning, and image/text reconstruction
regularization to learn fine-grained visual features while preserving global
semantic alignment. Our approach, scaling to over 1B parameters, outperforms
existing state-of-the-art (SOTA) models across multiple benchmarks,
establishing a new SOTA zero-shot performance on ImageNet-1K, delivering up to
a 2times enhancement over SigLIP on RxRx1 in linear probing for few-shot
classification, and improving vision-language models, achieving over 3times
higher scores than SigLIP on MMVP. Our code/checkpoints are available at
https://tulip-berkeley.github.io
TULIP enhances image-text contrastive models by incorporating generative data augmentation and contrastive learning to improve fine-grained visual features and zero-shot performance.
CLIP SigLIP vision-centric tasks high-fidelity image understanding counting depth estimation fine-grained object recognition language alignment generative data augmentation image-image contrastive learning text-text contrastive learning image/text reconstruction regularization ImageNet-1K RxRx1 zero-shot performance few-shot classification vision-language models MMVP