Improved baselines for vision-language pre-training
Contrastive learning has emerged as an efficient framework to learn
multimodal representations. CLIP, a seminal work in this area, achieved
impressive results by training on paired image-text data using the contrastive
loss. Recent work claims improvements over CLIP using additional
non-contrastive losses inspired from self-supervised learning. However, it is
sometimes hard to disentangle the contribution of these additional losses from
other implementation details, e.g., data augmentation or regularization
techniques, used to train the model. To shed light on this matter, in this
paper, we first propose, implement and evaluate several baselines obtained by
combining contrastive learning with recent advances in self-supervised
learning. In particular, we use the loss functions that were proven successful
for visual self-supervised learning to align image and text modalities. We find
that these baselines outperform a basic implementation of CLIP. However, when a
stronger training recipe is employed, the advantage disappears. Indeed, we find
that a simple CLIP baseline can also be improved substantially, up to a 25%
relative improvement on downstream zero-shot tasks, by using well-known
training techniques that are popular in other subfields. Moreover, we discover
that it is enough to apply image and text augmentations to make up for most of
the improvement attained by prior works. With our improved training recipe for
CLIP, we obtain state-of-the-art performance on four standard datasets, and
consistently outperform prior work (up to +4% on the largest dataset), while
being substantially simpler.
64630fa9960c80d19c67d403
Enrico Fini
claimed_verified
2023-05-16T13:59:16.748Z
639f6ebb9f1f2baab2f69513
/avatars/adb9d4feaf0a6184559b049bfb96ca0d.svg
Enrico Fini
user
efini
64630fa9960c80d19c67d404
Pietro Astolfi
claimed_verified
2023-05-16T15:31:36.780Z
646388dfd4d34b01f45d87c3
/avatars/a66bf45e780d0a82d00035f16c24d4c8.svg
Pietro Astolfi
user
pietroastolfi
64630fa9960c80d19c67d405
Adriana Romero-Soriano
64630fa9960c80d19c67d406
Jakob Verbeek
64630fa9960c80d19c67d407
Michal Drozdzal
Improved baselines for vision-language pre-training
Contrastive learning has emerged as an efficient framework to learn
multimodal representations. CLIP, a seminal work in this area, achieved
impressive results by training on paired image-text data using the contrastive
loss. Recent work claims improvements over CLIP using additional
non-contrastive losses inspired from self-supervised learning. However, it is
sometimes hard to disentangle the contribution of these additional losses from
other implementation details, e.g., data augmentation or regularization
techniques, used to train the model. To shed light on this matter, in this
paper, we first propose, implement and evaluate several baselines obtained by
combining contrastive learning with recent advances in self-supervised
learning. In particular, we use the loss functions that were proven successful
for visual self-supervised learning to align image and text modalities. We find
that these baselines outperform a basic implementation of CLIP. However, when a
stronger training recipe is employed, the advantage disappears. Indeed, we find
that a simple CLIP baseline can also be improved substantially, up to a 25%
relative improvement on downstream zero-shot tasks, by using well-known
training techniques that are popular in other subfields. Moreover, we discover
that it is enough to apply image and text augmentations to make up for most of
the improvement attained by prior works. With our improved training recipe for
CLIP, we obtain state-of-the-art performance on four standard datasets, and
consistently outperform prior work (up to +4% on the largest dataset), while
being substantially simpler.
Improved training techniques and augmentations can effectively boost CLIP's performance on multimodal tasks, surpassing models with additional non-contrastive losses.
contrastive learning CLIP contrastive loss self-supervised learning image-text alignment data augmentation regularization techniques zero-shot tasks