title: Demystifying CLIP Data
summary: Contrastive Language-Image Pre-training (CLIP) is an approach that has
  advanced research and applications in computer vision, fueling modern
  recognition systems and generative models. We believe that the main ingredient
  to the success of CLIP is its data and not the model architecture or
  pre-training objective. However, CLIP only provides very limited information
  about its data and how it has been collected, leading to works that aim to
  reproduce CLIP's data by filtering with its model parameters. In this work, we
  intend to reveal CLIP's data curation approach and in our pursuit of making it
  open to the community introduce Metadata-Curated Language-Image Pre-training
  (MetaCLIP). MetaCLIP takes a raw data pool and metadata (derived from CLIP's
  concepts) and yields a balanced subset over the metadata distribution. Our
  experimental study rigorously isolates the model and training settings,
  concentrating solely on data. MetaCLIP applied to CommonCrawl with 400M
  image-text data pairs outperforms CLIP's data on multiple standard benchmarks.
  In zero-shot ImageNet classification, MetaCLIP achieves 70.8% accuracy,
  surpassing CLIP's 68.3% on ViT-B models. Scaling to 1B data, while maintaining
  the same training budget, attains 72.4%. Our observations hold across various
  model sizes, exemplified by ViT-H achieving 80.5%, without any
  bells-and-whistles. Curation code and training data distribution on metadata is
  made available at https://github.com/facebookresearch/MetaCLIP.
paper_authors._id: 65166a9001d873dd191cf4ba
paper_authors.name: Hu Xu
paper_authors._id: 65166a9001d873dd191cf4bb
paper_authors.name: Saining Xie
paper_authors._id: 65166a9001d873dd191cf4bc
paper_authors.name: Xiaoqing Ellen Tan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-02T06:58:58.360Z
paper_authors._id: 6345ab8ffe134dfd7a0d1059
paper_authors.avatarUrl: /avatars/159a364ac4f8a2ed001346d4cbb7f2e3.svg
paper_authors.fullname: Ellen Tan
paper_authors.type: user
paper_authors.user: ellenxtan
paper_authors._id: 65166a9001d873dd191cf4bd
paper_authors.name: Po-Yao Huang
paper_authors._id: 65166a9001d873dd191cf4be
paper_authors.name: Russell Howes
paper_authors._id: 65166a9001d873dd191cf4bf
paper_authors.name: Vasu Sharma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-10T08:22:32.137Z
paper_authors._id: 65492730a3d1682f79f3ab7a
paper_authors.avatarUrl: /avatars/fc62605bdf10947dab393305c51ace96.svg
paper_authors.fullname: Vasu Sharma
paper_authors.type: user
paper_authors.user: vasusharma55
paper_authors._id: 65166a9001d873dd191cf4c0
paper_authors.name: Shang-Wen Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-28T19:31:40.914Z
paper_authors._id: 64e3a7b8e12618b261fff364
paper_authors.avatarUrl: /avatars/860a5011f150537eb923f76d354f1bc5.svg
paper_authors.fullname: Dan Li
paper_authors.type: user
paper_authors.user: swdanielli
paper_authors._id: 65166a9001d873dd191cf4c1
paper_authors.name: Gargi Ghosh
paper_authors._id: 65166a9001d873dd191cf4c2
paper_authors.name: Luke Zettlemoyer
paper_authors._id: 65166a9001d873dd191cf4c3
paper_authors.name: Christoph Feichtenhofer
paper_title: Demystifying CLIP Data
paper_summary: Contrastive Language-Image Pre-training (CLIP) is an approach that has
  advanced research and applications in computer vision, fueling modern
  recognition systems and generative models. We believe that the main ingredient
  to the success of CLIP is its data and not the model architecture or
  pre-training objective. However, CLIP only provides very limited information
  about its data and how it has been collected, leading to works that aim to
  reproduce CLIP's data by filtering with its model parameters. In this work, we
  intend to reveal CLIP's data curation approach and in our pursuit of making it
  open to the community introduce Metadata-Curated Language-Image Pre-training
  (MetaCLIP). MetaCLIP takes a raw data pool and metadata (derived from CLIP's
  concepts) and yields a balanced subset over the metadata distribution. Our
  experimental study rigorously isolates the model and training settings,
  concentrating solely on data. MetaCLIP applied to CommonCrawl with 400M
  image-text data pairs outperforms CLIP's data on multiple standard benchmarks.
  In zero-shot ImageNet classification, MetaCLIP achieves 70.8% accuracy,
  surpassing CLIP's 68.3% on ViT-B models. Scaling to 1B data, while maintaining
  the same training budget, attains 72.4%. Our observations hold across various
  model sizes, exemplified by ViT-H achieving 80.5%, without any
  bells-and-whistles. Curation code and training data distribution on metadata is
  made available at https://github.com/facebookresearch/MetaCLIP.
paper_ai_keywords: Contrastive Language-Image Pre-training (CLIP) Metadata-Curated Language-Image Pre-training (MetaCLIP) zero-shot ImageNet classification ViT-B models ViT-H models