Image Captioners Are Scalable Vision Learners Too
Contrastive pretraining on image-text pairs from the web is one of the most
popular large-scale pretraining strategies for vision backbones, especially in
the context of large multimodal models. At the same time, image captioning on
this type of data is commonly considered an inferior pretraining strategy. In
this paper, we perform a fair comparison of these two pretraining strategies,
carefully matching training data, compute, and model capacity. Using a standard
encoder-decoder transformer, we find that captioning alone is surprisingly
effective: on classification tasks, captioning produces vision encoders
competitive with contrastively pretrained encoders, while surpassing them on
vision & language tasks. We further analyze the effect of the model
architecture and scale, as well as the pretraining data on the representation
quality, and find that captioning exhibits the same or better scaling behavior
along these axes. Overall our results show that plain image captioning is a
more powerful pretraining strategy than was previously believed.
6489312e9ab28735f19eff7c
Michael Tschannen
claimed_verified
2023-06-14T12:15:11.324Z
6489893e1ec8356ba5bb9777
/avatars/54354c1e5774cadd1d83d42054e9d96b.svg
Michael Tschannen
user
mitsch
6489312e9ab28735f19eff7d
Manoj Kumar
claimed_verified
2024-05-27T07:23:08.340Z
6654321cfdf911031c53347a
/avatars/a23d4b5b6d456618f18a87aba4ecd822.svg
Manoj Kumar
user
mechcoder
6489312e9ab28735f19eff7e
Andreas Steiner
6489312e9ab28735f19eff7f
Xiaohua Zhai
6489312e9ab28735f19eff80
Neil Houlsby
claimed_verified
2023-07-03T15:23:05.878Z
64a2c7f263f69fb98d3bfdb4
/avatars/a362a236c0654b7605dcb7673e309335.svg
Neil Houlsby
user
neilhoulsby
6489312e9ab28735f19eff81
Lucas Beyer
claimed_verified
2024-01-09T14:26:23.485Z
642d334ff65714b4585f2de4
https://cdn-avatars.huggingface.co/v1/production/uploads/642d334ff65714b4585f2de4/gxBynq5KyoUP0VlAQD3-w.jpeg
Lucas Beyer
user
giffmana
Image Captioners Are Scalable Vision Learners Too
Contrastive pretraining on image-text pairs from the web is one of the most
popular large-scale pretraining strategies for vision backbones, especially in
the context of large multimodal models. At the same time, image captioning on
this type of data is commonly considered an inferior pretraining strategy. In
this paper, we perform a fair comparison of these two pretraining strategies,
carefully matching training data, compute, and model capacity. Using a standard
encoder-decoder transformer, we find that captioning alone is surprisingly
effective: on classification tasks, captioning produces vision encoders
competitive with contrastively pretrained encoders, while surpassing them on
vision & language tasks. We further analyze the effect of the model
architecture and scale, as well as the pretraining data on the representation
quality, and find that captioning exhibits the same or better scaling behavior
along these axes. Overall our results show that plain image captioning is a
more powerful pretraining strategy than was previously believed.
Captioning alone, when done with a carefully controlled comparison, proves to be as effective and sometimes more powerful than contrastive pretraining in vision and vision-language tasks.
contrastive pretraining image-text pairs vision backbones multimodal models image captioning encoder-decoder transformer vision encoders classification tasks vision & language tasks representation quality scaling behavior