title: EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameters
summary: Scaling up contrastive language-image pretraining (CLIP) is critical for
  empowering both vision and multimodal models. We present EVA-CLIP-18B, the
  largest and most powerful open-source CLIP model to date, with 18-billion
  parameters. With only 6-billion training samples seen, EVA-CLIP-18B achieves an
  exceptional 80.7% zero-shot top-1 accuracy averaged across 27 widely recognized
  image classification benchmarks, outperforming its forerunner EVA-CLIP
  (5-billion parameters) and other open-source CLIP models by a large margin.
  Remarkably, we observe a consistent performance improvement with the model size
  scaling of EVA-CLIP, despite maintaining a constant training dataset of
  2-billion image-text pairs from LAION-2B and COYO-700M. This dataset is openly
  available and much smaller than the in-house datasets (e.g., DFN-5B, WebLI-10B)
  employed in other state-of-the-art CLIP models. EVA-CLIP-18B demonstrates the
  potential of EVA-style weak-to-strong visual model scaling. With our model
  weights made publicly available, we hope to facilitate future research in
  vision and multimodal foundation models.
paper_authors._id: 65c2e6a7400d860a3ea74024
paper_authors.name: Quan Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T09:11:53.801Z
paper_authors._id: 630d7a8f81ef9b1772b67f4c
paper_authors.avatarUrl: /avatars/00757abd6e548ccebb5bfb233be129a2.svg
paper_authors.fullname: Quan Sun
paper_authors.type: user
paper_authors.user: QuanSun
paper_authors._id: 65c2e6a7400d860a3ea74025
paper_authors.name: Jinsheng Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T09:12:00.529Z
paper_authors._id: 6444a987af034cdfd69f6f8d
paper_authors.avatarUrl: /avatars/36544a102771a80c4fc6cb39e0973b2f.svg
paper_authors.fullname: Jinsheng Wang
paper_authors.type: user
paper_authors.user: wolfwjs
paper_authors._id: 65c2e6a7400d860a3ea74026
paper_authors.name: Qiying Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T09:12:07.089Z
paper_authors._id: 6341375be5071b0c5cc946a3
paper_authors.avatarUrl: /avatars/c26a32b54a3f7731854059d247b5c523.svg
paper_authors.fullname: Qiying Yu
paper_authors.type: user
paper_authors.user: qiying
paper_authors._id: 65c2e6a7400d860a3ea74027
paper_authors.name: Yufeng Cui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T09:12:13.155Z
paper_authors._id: 648683de623b5f050213f2be
paper_authors.avatarUrl: /avatars/83ecbcf4a21f68d2893de79f0444d6e3.svg
paper_authors.fullname: Yufeng Cui
paper_authors.type: user
paper_authors.user: YufengCui
paper_authors._id: 65c2e6a7400d860a3ea74028
paper_authors.name: Fan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T07:22:53.816Z
paper_authors._id: 640ed40dc025ddf618950af7
paper_authors.avatarUrl: /avatars/a99abd4346fa649ad4144f284ebcc972.svg
paper_authors.fullname: Fan Zhang
paper_authors.type: user
paper_authors.user: ryanzhangfan
paper_authors._id: 65c2e6a7400d860a3ea74029
paper_authors.name: Xiaosong Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T09:13:05.307Z
paper_authors._id: 64e473f0b78bc92221ab1883
paper_authors.avatarUrl: /avatars/85dd02c2b4879d7696b552f0f9dc2680.svg
paper_authors.fullname: Xiaosong Zhang
paper_authors.type: user
paper_authors.user: zhangxiaosong18
paper_authors._id: 65c2e6a7400d860a3ea7402a
paper_authors.name: Xinlong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T09:13:12.906Z
paper_authors._id: 63ca558304c979828311c5a5
paper_authors.avatarUrl: /avatars/2a439d79fba2f987cabe780d10c94d25.svg
paper_authors.fullname: Xinlong Wang
paper_authors.type: user
paper_authors.user: xinlongwang
paper_title: EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameters
paper_summary: Scaling up contrastive language-image pretraining (CLIP) is critical for
  empowering both vision and multimodal models. We present EVA-CLIP-18B, the
  largest and most powerful open-source CLIP model to date, with 18-billion
  parameters. With only 6-billion training samples seen, EVA-CLIP-18B achieves an
  exceptional 80.7% zero-shot top-1 accuracy averaged across 27 widely recognized
  image classification benchmarks, outperforming its forerunner EVA-CLIP
  (5-billion parameters) and other open-source CLIP models by a large margin.
  Remarkably, we observe a consistent performance improvement with the model size
  scaling of EVA-CLIP, despite maintaining a constant training dataset of
  2-billion image-text pairs from LAION-2B and COYO-700M. This dataset is openly
  available and much smaller than the in-house datasets (e.g., DFN-5B, WebLI-10B)
  employed in other state-of-the-art CLIP models. EVA-CLIP-18B demonstrates the
  potential of EVA-style weak-to-strong visual model scaling. With our model
  weights made publicly available, we hope to facilitate future research in
  vision and multimodal foundation models.
paper_ai_keywords: contrastive language-image pretraining CLIP EVA-CLIP-18B parameters zero-shot top-1 accuracy image classification benchmarks EVA-CLIP visual model scaling weak-to-strong visual model scaling