Demystifying CLIP Data
Contrastive Language-Image Pre-training (CLIP) is an approach that has
advanced research and applications in computer vision, fueling modern
recognition systems and generative models. We believe that the main ingredient
to the success of CLIP is its data and not the model architecture or
pre-training objective. However, CLIP only provides very limited information
about its data and how it has been collected, leading to works that aim to
reproduce CLIP's data by filtering with its model parameters. In this work, we
intend to reveal CLIP's data curation approach and in our pursuit of making it
open to the community introduce Metadata-Curated Language-Image Pre-training
(MetaCLIP). MetaCLIP takes a raw data pool and metadata (derived from CLIP's
concepts) and yields a balanced subset over the metadata distribution. Our
experimental study rigorously isolates the model and training settings,
concentrating solely on data. MetaCLIP applied to CommonCrawl with 400M
image-text data pairs outperforms CLIP's data on multiple standard benchmarks.
In zero-shot ImageNet classification, MetaCLIP achieves 70.8% accuracy,
surpassing CLIP's 68.3% on ViT-B models. Scaling to 1B data, while maintaining
the same training budget, attains 72.4%. Our observations hold across various
model sizes, exemplified by ViT-H achieving 80.5%, without any
bells-and-whistles. Curation code and training data distribution on metadata is
made available at https://github.com/facebookresearch/MetaCLIP.
65166a9001d873dd191cf4ba
Hu Xu
65166a9001d873dd191cf4bb
Saining Xie
65166a9001d873dd191cf4bc
Xiaoqing Ellen Tan
admin_assigned
2023-10-02T06:58:58.360Z
6345ab8ffe134dfd7a0d1059
/avatars/159a364ac4f8a2ed001346d4cbb7f2e3.svg
Ellen Tan
user
ellenxtan
65166a9001d873dd191cf4bd
Po-Yao Huang
65166a9001d873dd191cf4be
Russell Howes
65166a9001d873dd191cf4bf
Vasu Sharma
claimed_verified
2025-01-10T08:22:32.137Z
65492730a3d1682f79f3ab7a
/avatars/fc62605bdf10947dab393305c51ace96.svg
Vasu Sharma
user
vasusharma55
65166a9001d873dd191cf4c0
Shang-Wen Li
claimed_verified
2024-04-28T19:31:40.914Z
64e3a7b8e12618b261fff364
/avatars/860a5011f150537eb923f76d354f1bc5.svg
Shang-Wen Daniel Li
user
swdanielli
65166a9001d873dd191cf4c1
Gargi Ghosh
65166a9001d873dd191cf4c2
Luke Zettlemoyer
65166a9001d873dd191cf4c3
Christoph Feichtenhofer
Demystifying CLIP Data
Contrastive Language-Image Pre-training (CLIP) is an approach that has
advanced research and applications in computer vision, fueling modern
recognition systems and generative models. We believe that the main ingredient
to the success of CLIP is its data and not the model architecture or
pre-training objective. However, CLIP only provides very limited information
about its data and how it has been collected, leading to works that aim to
reproduce CLIP's data by filtering with its model parameters. In this work, we
intend to reveal CLIP's data curation approach and in our pursuit of making it
open to the community introduce Metadata-Curated Language-Image Pre-training
(MetaCLIP). MetaCLIP takes a raw data pool and metadata (derived from CLIP's
concepts) and yields a balanced subset over the metadata distribution. Our
experimental study rigorously isolates the model and training settings,
concentrating solely on data. MetaCLIP applied to CommonCrawl with 400M
image-text data pairs outperforms CLIP's data on multiple standard benchmarks.
In zero-shot ImageNet classification, MetaCLIP achieves 70.8% accuracy,
surpassing CLIP's 68.3% on ViT-B models. Scaling to 1B data, while maintaining
the same training budget, attains 72.4%. Our observations hold across various
model sizes, exemplified by ViT-H achieving 80.5%, without any
bells-and-whistles. Curation code and training data distribution on metadata is
made available at https://github.com/facebookresearch/MetaCLIP.
MetaCLIP, a metadata-driven data curation approach for language-image pre-training, outperforms CLIP and scales effectively with larger datasets.
Contrastive Language-Image Pre-training CLIP Metadata-Curated Language-Image Pre-training MetaCLIP raw data pool metadata distribution zero-shot ImageNet classification ViT-B ViT-H