title: Should VLMs be Pre-trained with Image Data?
summary: Pre-trained LLMs that are further trained with image data perform well on
  vision-language tasks. While adding images during a second training phase
  effectively unlocks this capability, it is unclear how much of a gain or loss
  this two-step pipeline gives over VLMs which integrate images earlier into the
  training process. To investigate this, we train models spanning various
  datasets, scales, image-text ratios, and amount of pre-training done before
  introducing vision tokens. We then fine-tune these models and evaluate their
  downstream performance on a suite of vision-language and text-only tasks. We
  find that pre-training with a mixture of image and text data allows models to
  perform better on vision-language tasks while maintaining strong performance on
  text-only evaluations. On an average of 6 diverse tasks, we find that for a 1B
  model, introducing visual tokens 80% of the way through pre-training results in
  a 2% average improvement over introducing visual tokens to a fully pre-trained
  model.
paper_authors._id: 67cfc310f2b1fe815dc24ebf
paper_authors.name: Sedrick Keh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T14:52:53.915Z
paper_authors._id: 6365c1158b6bd0d9f6605d3d
paper_authors.avatarUrl: /avatars/1e3e6573e0cb2bb1b9d8043045120ee6.svg
paper_authors.fullname: Sedrick Keh
paper_authors.type: user
paper_authors.user: sedrickkeh
paper_authors._id: 67cfc310f2b1fe815dc24ec0
paper_authors.name: Jean Mercat
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T14:53:00.187Z
paper_authors._id: 62be399f01dc22b4d22fc990
paper_authors.avatarUrl: /avatars/3e1f3ed30773cbb410b8a9d3c854cac7.svg
paper_authors.fullname: Jean Mercat
paper_authors.type: user
paper_authors.user: jmercat
paper_authors._id: 67cfc310f2b1fe815dc24ec1
paper_authors.name: Samir Yitzhak Gadre
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T14:53:10.097Z
paper_authors._id: 62be1cd77c0388414f012f23
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656626356583-noauth.jpeg
paper_authors.fullname: Samir Gadre
paper_authors.type: user
paper_authors.user: samirg
paper_authors._id: 67cfc310f2b1fe815dc24ec2
paper_authors.name: Kushal Arora
paper_authors._id: 67cfc310f2b1fe815dc24ec3
paper_authors.name: Igor Vasiljevic
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T14:54:33.031Z
paper_authors._id: 633b2d317af633cbcd03a53e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/633b2d317af633cbcd03a53e/awtbCqUE4qBusxuu17iFg.png
paper_authors.fullname: Igor Vasiljevic
paper_authors.type: user
paper_authors.user: ivas-tri
paper_authors._id: 67cfc310f2b1fe815dc24ec4
paper_authors.name: Benjamin Burchfiel
paper_authors._id: 67cfc310f2b1fe815dc24ec5
paper_authors.name: Shuran Song
paper_authors._id: 67cfc310f2b1fe815dc24ec6
paper_authors.name: Russ Tedrake
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T14:53:34.722Z
paper_authors._id: 668924108c440fe1951e2d2a
paper_authors.avatarUrl: /avatars/97e60f1ecb0b7d8f7238a4e92944e370.svg
paper_authors.fullname: Russ Tedrake
paper_authors.type: user
paper_authors.user: RussTedrake
paper_authors._id: 67cfc310f2b1fe815dc24ec7
paper_authors.name: Thomas Kollar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T14:53:47.408Z
paper_authors._id: 64358e662d0ed796668cb438
paper_authors.avatarUrl: /avatars/6f86547d8691e061e289dd906deedfdb.svg
paper_authors.fullname: Thomas Kollar
paper_authors.type: user
paper_authors.user: thomas-kollar
paper_authors._id: 67cfc310f2b1fe815dc24ec8
paper_authors.name: Ludwig Schmidt
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T14:54:00.333Z
paper_authors._id: 6252375c818a5dc29ab9d945
paper_authors.avatarUrl: /avatars/cd4f0b220ea36e13ff71157cffe9d9d1.svg
paper_authors.fullname: Ludwig Schmidt
paper_authors.type: user
paper_authors.user: ludwigschmidt
paper_authors._id: 67cfc310f2b1fe815dc24ec9
paper_authors.name: Achal Dave
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-11T14:54:08.423Z
paper_authors._id: 6488e29b78570cedad71eb2d
paper_authors.avatarUrl: /avatars/0d8bb52c1e2be8d6aa7c732cf92fc5a9.svg
paper_authors.fullname: Achal Dave
paper_authors.type: user
paper_authors.user: achal-tri
paper_title: Should VLMs be Pre-trained with Image Data?
paper_summary: Pre-trained LLMs that are further trained with image data perform well on
  vision-language tasks. While adding images during a second training phase
  effectively unlocks this capability, it is unclear how much of a gain or loss
  this two-step pipeline gives over VLMs which integrate images earlier into the
  training process. To investigate this, we train models spanning various
  datasets, scales, image-text ratios, and amount of pre-training done before
  introducing vision tokens. We then fine-tune these models and evaluate their
  downstream performance on a suite of vision-language and text-only tasks. We
  find that pre-training with a mixture of image and text data allows models to
  perform better on vision-language tasks while maintaining strong performance on
  text-only evaluations. On an average of 6 diverse tasks, we find that for a 1B
  model, introducing visual tokens 80% of the way through pre-training results in
  a 2% average improvement over introducing visual tokens to a fully pre-trained
  model.
paper_ai_keywords: pre-trained LLMs vision-language tasks fine-tune vision tokens