title: Improving CLIP Training with Language Rewrites
summary: Contrastive Language-Image Pre-training (CLIP) stands as one of the most
  effective and scalable methods for training transferable vision models using
  paired image and text data. CLIP models are trained using contrastive loss,
  which typically relies on data augmentations to prevent overfitting and
  shortcuts. However, in the CLIP training paradigm, data augmentations are
  exclusively applied to image inputs, while language inputs remain unchanged
  throughout the entire training process, limiting the exposure of diverse texts
  to the same image. In this paper, we introduce Language augmented CLIP
  (LaCLIP), a simple yet highly effective approach to enhance CLIP training
  through language rewrites. Leveraging the in-context learning capability of
  large language models, we rewrite the text descriptions associated with each
  image. These rewritten texts exhibit diversity in sentence structure and
  vocabulary while preserving the original key concepts and meanings. During
  training, LaCLIP randomly selects either the original texts or the rewritten
  versions as text augmentations for each image. Extensive experiments on CC3M,
  CC12M, RedCaps and LAION-400M datasets show that CLIP pre-training with
  language rewrites significantly improves the transfer performance without
  computation or memory overhead during training. Specifically for ImageNet
  zero-shot accuracy, LaCLIP outperforms CLIP by 8.2% on CC12M and 2.4% on
  LAION-400M. Code is available at https://github.com/LijieFan/LaCLIP.
paper_authors._id: 64780dbf312e3909019a8766
paper_authors.name: Lijie Fan
paper_authors._id: 64780dbf312e3909019a8767
paper_authors.name: Dilip Krishnan
paper_authors._id: 64780dbf312e3909019a8768
paper_authors.name: Phillip Isola
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-01T08:37:01.067Z
paper_authors._id: 623cb1f3ca87989bbf797eee
paper_authors.avatarUrl: /avatars/13fbd5b123b8676561751f820317600c.svg
paper_authors.fullname: Phillip Isola
paper_authors.type: user
paper_authors.user: phillipi
paper_authors._id: 64780dbf312e3909019a8769
paper_authors.name: Dina Katabi
paper_authors._id: 64780dbf312e3909019a876a
paper_authors.name: Yonglong Tian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-02T07:16:38.600Z
paper_authors._id: 64793086ac77bb7603842456
paper_authors.avatarUrl: /avatars/5891292138d372a9cc063f86c08b8174.svg
paper_authors.fullname: Yonglong Tian
paper_authors.type: user
paper_authors.user: tyl5566
paper_title: Improving CLIP Training with Language Rewrites
paper_summary: Contrastive Language-Image Pre-training (CLIP) stands as one of the most
  effective and scalable methods for training transferable vision models using
  paired image and text data. CLIP models are trained using contrastive loss,
  which typically relies on data augmentations to prevent overfitting and
  shortcuts. However, in the CLIP training paradigm, data augmentations are
  exclusively applied to image inputs, while language inputs remain unchanged
  throughout the entire training process, limiting the exposure of diverse texts
  to the same image. In this paper, we introduce Language augmented CLIP
  (LaCLIP), a simple yet highly effective approach to enhance CLIP training
  through language rewrites. Leveraging the in-context learning capability of
  large language models, we rewrite the text descriptions associated with each
  image. These rewritten texts exhibit diversity in sentence structure and
  vocabulary while preserving the original key concepts and meanings. During
  training, LaCLIP randomly selects either the original texts or the rewritten
  versions as text augmentations for each image. Extensive experiments on CC3M,
  CC12M, RedCaps and LAION-400M datasets show that CLIP pre-training with
  language rewrites significantly improves the transfer performance without
  computation or memory overhead during training. Specifically for ImageNet
  zero-shot accuracy, LaCLIP outperforms CLIP by 8.2% on CC12M and 2.4% on
  LAION-400M. Code is available at https://github.com/LijieFan/LaCLIP.
paper_ai_summary: Language augmented CLIP (LaCLIP) enhances CLIP training by introducing text rewrites, improving zero-shot accuracy without increasing computational costs.
paper_ai_keywords: Contrastive Language-Image Pre-training CLIP contrastive loss data augmentations image inputs language inputs LaCLIP in-context learning large language models text rewrites CC3M CC12M RedCaps LAION-400M ImageNet zero-shot accuracy