title: An Inverse Scaling Law for CLIP Training
summary: CLIP, the first foundation model that connects images and text, has enabled
  many recent breakthroughs in computer vision. However, its associated training
  cost is prohibitively high, imposing a significant barrier to its widespread
  exploration. In this paper, we present a surprising finding that there exists
  an inverse scaling law for CLIP training, whereby the larger the image/text
  encoders used, the shorter the sequence length of image/text tokens that can be
  applied in training. Moreover, we showcase that the strategy for reducing
  image/text token length plays a crucial role in determining the quality of this
  scaling law.
    As a result of this finding, we are able to successfully train CLIP even by
  using academic resources. For example, on an A100 eight-GPU server, our CLIP
  models achieve zero-shot top-1 ImageNet accuracies of 63.2% in ~2 days, 67.8%
  in ~3 days, and 69.3% in ~4 days. By reducing the computation barrier
  associated with CLIP, we hope to inspire more research in this field,
  particularly from academics. Our code is available at
  https://github.com/UCSC-VLAA/CLIPA.
paper_authors._id: 645ddb74363a956edb13b363
paper_authors.name: Xianhang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-12T17:13:25.211Z
paper_authors._id: 645e72d4b5c9a8666d0ddc1c
paper_authors.avatarUrl: /avatars/f0f1684731eb5ae148eadf841cceb18e.svg
paper_authors.fullname: xhl
paper_authors.type: user
paper_authors.user: Xianhang
paper_authors._id: 645ddb74363a956edb13b364
paper_authors.name: Zeyu Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-16T09:55:45.516Z
paper_authors._id: 6462879777f63463fc5f75f9
paper_authors.avatarUrl: /avatars/f19d14a009e940e3b565a6c62251599b.svg
paper_authors.fullname: Zeyu Wang
paper_authors.type: user
paper_authors.user: zw123
paper_authors._id: 645ddb74363a956edb13b365
paper_authors.name: Cihang Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-14T17:50:57.213Z
paper_authors._id: 645eb61da3c5cd8a16efffff
paper_authors.avatarUrl: /avatars/9112bfeed598dfabf9e077e69e09ecc9.svg
paper_authors.fullname: Cihang Xie
paper_authors.type: user
paper_authors.user: cihangxie
paper_title: An Inverse Scaling Law for CLIP Training
paper_summary: CLIP, the first foundation model that connects images and text, has enabled
  many recent breakthroughs in computer vision. However, its associated training
  cost is prohibitively high, imposing a significant barrier to its widespread
  exploration. In this paper, we present a surprising finding that there exists
  an inverse scaling law for CLIP training, whereby the larger the image/text
  encoders used, the shorter the sequence length of image/text tokens that can be
  applied in training. Moreover, we showcase that the strategy for reducing
  image/text token length plays a crucial role in determining the quality of this
  scaling law.
    As a result of this finding, we are able to successfully train CLIP even by
  using academic resources. For example, on an A100 eight-GPU server, our CLIP
  models achieve zero-shot top-1 ImageNet accuracies of 63.2% in ~2 days, 67.8%
  in ~3 days, and 69.3% in ~4 days. By reducing the computation barrier
  associated with CLIP, we hope to inspire more research in this field,
  particularly from academics. Our code is available at
  https://github.com/UCSC-VLAA/CLIPA.
paper_ai_keywords: CLIP foundation model inverse scaling law image/text encoders token length zero-shot top-1 ImageNet accuracies