title: Improved baselines for vision-language pre-training
summary: Contrastive learning has emerged as an efficient framework to learn
  multimodal representations. CLIP, a seminal work in this area, achieved
  impressive results by training on paired image-text data using the contrastive
  loss. Recent work claims improvements over CLIP using additional
  non-contrastive losses inspired from self-supervised learning. However, it is
  sometimes hard to disentangle the contribution of these additional losses from
  other implementation details, e.g., data augmentation or regularization
  techniques, used to train the model. To shed light on this matter, in this
  paper, we first propose, implement and evaluate several baselines obtained by
  combining contrastive learning with recent advances in self-supervised
  learning. In particular, we use the loss functions that were proven successful
  for visual self-supervised learning to align image and text modalities. We find
  that these baselines outperform a basic implementation of CLIP. However, when a
  stronger training recipe is employed, the advantage disappears. Indeed, we find
  that a simple CLIP baseline can also be improved substantially, up to a 25%
  relative improvement on downstream zero-shot tasks, by using well-known
  training techniques that are popular in other subfields. Moreover, we discover
  that it is enough to apply image and text augmentations to make up for most of
  the improvement attained by prior works. With our improved training recipe for
  CLIP, we obtain state-of-the-art performance on four standard datasets, and
  consistently outperform prior work (up to +4% on the largest dataset), while
  being substantially simpler.
paper_authors._id: 64630fa9960c80d19c67d403
paper_authors.name: Enrico Fini
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-16T13:59:16.748Z
paper_authors._id: 639f6ebb9f1f2baab2f69513
paper_authors.avatarUrl: /avatars/adb9d4feaf0a6184559b049bfb96ca0d.svg
paper_authors.fullname: Enrico Fini
paper_authors.type: user
paper_authors.user: efini
paper_authors._id: 64630fa9960c80d19c67d404
paper_authors.name: Pietro Astolfi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-16T15:31:36.780Z
paper_authors._id: 646388dfd4d34b01f45d87c3
paper_authors.avatarUrl: /avatars/a66bf45e780d0a82d00035f16c24d4c8.svg
paper_authors.fullname: Pietro Astolfi
paper_authors.type: user
paper_authors.user: pietroastolfi
paper_authors._id: 64630fa9960c80d19c67d405
paper_authors.name: Adriana Romero-Soriano
paper_authors._id: 64630fa9960c80d19c67d406
paper_authors.name: Jakob Verbeek
paper_authors._id: 64630fa9960c80d19c67d407
paper_authors.name: Michal Drozdzal
paper_title: Improved baselines for vision-language pre-training
paper_summary: Contrastive learning has emerged as an efficient framework to learn
  multimodal representations. CLIP, a seminal work in this area, achieved
  impressive results by training on paired image-text data using the contrastive
  loss. Recent work claims improvements over CLIP using additional
  non-contrastive losses inspired from self-supervised learning. However, it is
  sometimes hard to disentangle the contribution of these additional losses from
  other implementation details, e.g., data augmentation or regularization
  techniques, used to train the model. To shed light on this matter, in this
  paper, we first propose, implement and evaluate several baselines obtained by
  combining contrastive learning with recent advances in self-supervised
  learning. In particular, we use the loss functions that were proven successful
  for visual self-supervised learning to align image and text modalities. We find
  that these baselines outperform a basic implementation of CLIP. However, when a
  stronger training recipe is employed, the advantage disappears. Indeed, we find
  that a simple CLIP baseline can also be improved substantially, up to a 25%
  relative improvement on downstream zero-shot tasks, by using well-known
  training techniques that are popular in other subfields. Moreover, we discover
  that it is enough to apply image and text augmentations to make up for most of
  the improvement attained by prior works. With our improved training recipe for
  CLIP, we obtain state-of-the-art performance on four standard datasets, and
  consistently outperform prior work (up to +4% on the largest dataset), while
  being substantially simpler.
paper_ai_summary: Improved training techniques and augmentations can effectively boost CLIP's performance on multimodal tasks, surpassing models with additional non-contrastive losses.
paper_ai_keywords: contrastive learning CLIP contrastive loss self-supervised learning image-text alignment data augmentation regularization techniques zero-shot tasks