title: MoDE: CLIP Data Experts via Clustering
summary: The success of contrastive language-image pretraining (CLIP) relies on the
  supervision from the pairing between images and captions, which tends to be
  noisy in web-crawled data. We present Mixture of Data Experts (MoDE) and learn
  a system of CLIP data experts via clustering. Each data expert is trained on
  one data cluster, being less sensitive to false negative noises in other
  clusters. At inference time, we ensemble their outputs by applying weights
  determined through the correlation between task metadata and cluster
  conditions. To estimate the correlation precisely, the samples in one cluster
  should be semantically similar, but the number of data experts should still be
  reasonable for training and inference. As such, we consider the ontology in
  human language and propose to use fine-grained cluster centers to represent
  each data expert at a coarse-grained level. Experimental studies show that four
  CLIP data experts on ViT-B/16 outperform the ViT-L/14 by OpenAI CLIP and
  OpenCLIP on zero-shot image classification but with less (<35\%) training
  cost. Meanwhile, MoDE can train all data expert asynchronously and can flexibly
  include new data experts. The code is available at
  https://github.com/facebookresearch/MetaCLIP/tree/main/mode.
paper_authors._id: 662a9cab16435ebd17bf5e5f
paper_authors.name: Jiawei Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T12:10:10.328Z
paper_authors._id: 653b41de251b430ac072def1
paper_authors.avatarUrl: /avatars/cfefeca4e242f7e32c3db4fb7d9f812a.svg
paper_authors.fullname: Jiawei Ma
paper_authors.type: user
paper_authors.user: Phoenix1128
paper_authors._id: 662a9cab16435ebd17bf5e60
paper_authors.name: Po-Yao Huang
paper_authors._id: 662a9cab16435ebd17bf5e61
paper_authors.name: Saining Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T12:10:34.373Z
paper_authors._id: 6596422646624a86ff3b3bda
paper_authors.avatarUrl: /avatars/216e12b77e45ac5f1fa20932f5745411.svg
paper_authors.fullname: Saining Xie
paper_authors.type: user
paper_authors.user: sainx
paper_authors._id: 662a9cab16435ebd17bf5e62
paper_authors.name: Shang-Wen Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-28T19:30:16.890Z
paper_authors._id: 64e3a7b8e12618b261fff364
paper_authors.avatarUrl: /avatars/860a5011f150537eb923f76d354f1bc5.svg
paper_authors.fullname: Dan Li
paper_authors.type: user
paper_authors.user: swdanielli
paper_authors._id: 662a9cab16435ebd17bf5e63
paper_authors.name: Luke Zettlemoyer
paper_authors._id: 662a9cab16435ebd17bf5e64
paper_authors.name: Shih-Fu Chang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T12:11:05.711Z
paper_authors._id: 6616bd1fdb7e6d38160080c4
paper_authors.avatarUrl: /avatars/ba927a1c97e090b61eea9a3e04750692.svg
paper_authors.fullname: Jeff
paper_authors.type: user
paper_authors.user: shihfu
paper_authors._id: 662a9cab16435ebd17bf5e65
paper_authors.name: Wen-Tau Yih
paper_authors._id: 662a9cab16435ebd17bf5e66
paper_authors.name: Hu Xu
paper_title: MoDE: CLIP Data Experts via Clustering
paper_summary: The success of contrastive language-image pretraining (CLIP) relies on the
  supervision from the pairing between images and captions, which tends to be
  noisy in web-crawled data. We present Mixture of Data Experts (MoDE) and learn
  a system of CLIP data experts via clustering. Each data expert is trained on
  one data cluster, being less sensitive to false negative noises in other
  clusters. At inference time, we ensemble their outputs by applying weights
  determined through the correlation between task metadata and cluster
  conditions. To estimate the correlation precisely, the samples in one cluster
  should be semantically similar, but the number of data experts should still be
  reasonable for training and inference. As such, we consider the ontology in
  human language and propose to use fine-grained cluster centers to represent
  each data expert at a coarse-grained level. Experimental studies show that four
  CLIP data experts on ViT-B/16 outperform the ViT-L/14 by OpenAI CLIP and
  OpenCLIP on zero-shot image classification but with less (<35\%) training
  cost. Meanwhile, MoDE can train all data expert asynchronously and can flexibly
  include new data experts. The code is available at
  https://github.com/facebookresearch/MetaCLIP/tree/main/mode.
paper_ai_summary: Mixture of Data Experts (MoDE) enhances CLIP's performance by clustering data into experts, reducing noise and improving zero-shot classification with lower training costs.
paper_ai_keywords: contrastive language-image pretraining CLIP supervision data experts clustering false negative noises ensemble task metadata cluster conditions ontology fine-grained cluster centers coarse-grained level zero-shot image classification training cost asynchronous training