title: Image Captioners Are Scalable Vision Learners Too
summary: Contrastive pretraining on image-text pairs from the web is one of the most
  popular large-scale pretraining strategies for vision backbones, especially in
  the context of large multimodal models. At the same time, image captioning on
  this type of data is commonly considered an inferior pretraining strategy. In
  this paper, we perform a fair comparison of these two pretraining strategies,
  carefully matching training data, compute, and model capacity. Using a standard
  encoder-decoder transformer, we find that captioning alone is surprisingly
  effective: on classification tasks, captioning produces vision encoders
  competitive with contrastively pretrained encoders, while surpassing them on
  vision & language tasks. We further analyze the effect of the model
  architecture and scale, as well as the pretraining data on the representation
  quality, and find that captioning exhibits the same or better scaling behavior
  along these axes. Overall our results show that plain image captioning is a
  more powerful pretraining strategy than was previously believed.
paper_authors._id: 6489312e9ab28735f19eff7c
paper_authors.name: Michael Tschannen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-14T12:15:11.324Z
paper_authors._id: 6489893e1ec8356ba5bb9777
paper_authors.avatarUrl: /avatars/54354c1e5774cadd1d83d42054e9d96b.svg
paper_authors.fullname: Michael Tschannen
paper_authors.type: user
paper_authors.user: mitsch
paper_authors._id: 6489312e9ab28735f19eff7d
paper_authors.name: Manoj Kumar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-27T07:23:08.340Z
paper_authors._id: 6654321cfdf911031c53347a
paper_authors.avatarUrl: /avatars/a23d4b5b6d456618f18a87aba4ecd822.svg
paper_authors.fullname: Manoj Kumar
paper_authors.type: user
paper_authors.user: mechcoder
paper_authors._id: 6489312e9ab28735f19eff7e
paper_authors.name: Andreas Steiner
paper_authors._id: 6489312e9ab28735f19eff7f
paper_authors.name: Xiaohua Zhai
paper_authors._id: 6489312e9ab28735f19eff80
paper_authors.name: Neil Houlsby
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-03T15:23:05.878Z
paper_authors._id: 64a2c7f263f69fb98d3bfdb4
paper_authors.avatarUrl: /avatars/a362a236c0654b7605dcb7673e309335.svg
paper_authors.fullname: Neil Houlsby
paper_authors.type: user
paper_authors.user: neilhoulsby
paper_authors._id: 6489312e9ab28735f19eff81
paper_authors.name: Lucas Beyer
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-09T14:26:23.485Z
paper_authors._id: 642d334ff65714b4585f2de4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642d334ff65714b4585f2de4/gxBynq5KyoUP0VlAQD3-w.jpeg
paper_authors.fullname: Lucas Beyer
paper_authors.type: user
paper_authors.user: giffmana
paper_title: Image Captioners Are Scalable Vision Learners Too
paper_summary: Contrastive pretraining on image-text pairs from the web is one of the most
  popular large-scale pretraining strategies for vision backbones, especially in
  the context of large multimodal models. At the same time, image captioning on
  this type of data is commonly considered an inferior pretraining strategy. In
  this paper, we perform a fair comparison of these two pretraining strategies,
  carefully matching training data, compute, and model capacity. Using a standard
  encoder-decoder transformer, we find that captioning alone is surprisingly
  effective: on classification tasks, captioning produces vision encoders
  competitive with contrastively pretrained encoders, while surpassing them on
  vision & language tasks. We further analyze the effect of the model
  architecture and scale, as well as the pretraining data on the representation
  quality, and find that captioning exhibits the same or better scaling behavior
  along these axes. Overall our results show that plain image captioning is a
  more powerful pretraining strategy than was previously believed.
paper_ai_summary: Captioning alone, when done with a carefully controlled comparison, proves to be as effective and sometimes more powerful than contrastive pretraining in vision and vision-language tasks.
paper_ai_keywords: contrastive pretraining image-text pairs vision backbones multimodal models image captioning encoder-decoder transformer vision encoders classification tasks vision & language tasks representation quality scaling behavior