MetaCLIP 2: A Worldwide Scaling Recipe
Contrastive Language-Image Pretraining (CLIP) is a popular foundation model,
supporting from zero-shot classification, retrieval to encoders for multimodal
large language models (MLLMs). Although CLIP is successfully trained on
billion-scale image-text pairs from the English world, scaling CLIP's training
further to learning from the worldwide web data is still challenging: (1) no
curation method is available to handle data points from non-English world; (2)
the English performance from existing multilingual CLIP is worse than its
English-only counterpart, i.e., "curse of multilinguality" that is common in
LLMs. Here, we present MetaCLIP 2, the first recipe training CLIP from scratch
on worldwide web-scale image-text pairs. To generalize our findings, we conduct
rigorous ablations with minimal changes that are necessary to address the above
challenges and present a recipe enabling mutual benefits from English and
non-English world data. In zero-shot ImageNet classification, MetaCLIP 2
ViT-H/14 surpasses its English-only counterpart by 0.8% and mSigLIP by 0.7%,
and surprisingly sets new state-of-the-art without system-level confounding
factors (e.g., translation, bespoke architecture changes) on multilingual
benchmarks, such as CVQA with 57.4%, Babel-ImageNet with 50.2% and XM3600 with
64.3% on image-to-text retrieval.
688b6429292830d55fa68a2c
Yung-Sung Chuang
claimed_verified
2025-07-31T21:00:22.011Z
5df84571da6d0311fd3d5407
https://cdn-avatars.huggingface.co/v1/production/uploads/1650651305661-5df84571da6d0311fd3d5407.png
Yung-Sung Chuang
user
voidism
688b6429292830d55fa68a2d
Yang Li
688b6429292830d55fa68a2e
Dong Wang
688b6429292830d55fa68a2f
Ching-Feng Yeh
688b6429292830d55fa68a30
Kehan Lyu
688b6429292830d55fa68a31
Ramya Raghavendra
688b6429292830d55fa68a32
James Glass
688b6429292830d55fa68a33
Lifei Huang
688b6429292830d55fa68a34
Jason Weston
688b6429292830d55fa68a35
Luke Zettlemoyer
688b6429292830d55fa68a36
Xinlei Chen
688b6429292830d55fa68a37
Zhuang Liu
688b6429292830d55fa68a38
Saining Xie
688b6429292830d55fa68a39
Wen-tau Yih
688b6429292830d55fa68a3a
Shang-Wen Li
688b6429292830d55fa68a3b
Hu Xu
MetaCLIP 2: A Worldwide Scaling Recipe
Contrastive Language-Image Pretraining (CLIP) is a popular foundation model,
supporting from zero-shot classification, retrieval to encoders for multimodal
large language models (MLLMs). Although CLIP is successfully trained on
billion-scale image-text pairs from the English world, scaling CLIP's training
further to learning from the worldwide web data is still challenging: (1) no
curation method is available to handle data points from non-English world; (2)
the English performance from existing multilingual CLIP is worse than its
English-only counterpart, i.e., "curse of multilinguality" that is common in
LLMs. Here, we present MetaCLIP 2, the first recipe training CLIP from scratch
on worldwide web-scale image-text pairs. To generalize our findings, we conduct
rigorous ablations with minimal changes that are necessary to address the above
challenges and present a recipe enabling mutual benefits from English and
non-English world data. In zero-shot ImageNet classification, MetaCLIP 2
ViT-H/14 surpasses its English-only counterpart by 0.8% and mSigLIP by 0.7%,
and surprisingly sets new state-of-the-art without system-level confounding
factors (e.g., translation, bespoke architecture changes) on multilingual
benchmarks, such as CVQA with 57.4%, Babel-ImageNet with 50.2% and XM3600 with
64.3% on image-to-text retrieval.
MetaCLIP 2, trained on worldwide web-scale image-text pairs, improves zero-shot classification and multilingual benchmarks without system-level confounding factors.
Contrastive Language-Image Pretraining CLIP multilingual CLIP MetaCLIP 2 zero-shot classification image-text pairs multilingual benchmarks CVQA Babel-ImageNet XM3600 image-to-text retrieval