title: VILA: On Pre-training for Visual Language Models
summary: Visual language models (VLMs) rapidly progressed with the recent success of
  large language models. There have been growing efforts on visual instruction
  tuning to extend the LLM with visual inputs, but lacks an in-depth study of the
  visual language pre-training process, where the model learns to perform joint
  modeling on both modalities. In this work, we examine the design options for
  VLM pre-training by augmenting LLM towards VLM through step-by-step
  controllable comparisons. We introduce three main findings: (1) freezing LLMs
  during pre-training can achieve decent zero-shot performance, but lack
  in-context learning capability, which requires unfreezing the LLM; (2)
  interleaved pre-training data is beneficial whereas image-text pairs alone are
  not optimal; (3) re-blending text-only instruction data to image-text data
  during instruction fine-tuning not only remedies the degradation of text-only
  tasks, but also boosts VLM task accuracy. With an enhanced pre-training recipe
  we build VILA, a Visual Language model family that consistently outperforms the
  state-of-the-art models, e.g., LLaVA-1.5, across main benchmarks without bells
  and whistles. Multi-modal pre-training also helps unveil appealing properties
  of VILA, including multi-image reasoning, enhanced in-context learning, and
  better world knowledge.
paper_authors._id: 657920ef800326e825395dd1
paper_authors.name: Ji Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-14T10:34:52.393Z
paper_authors._id: 62f68e83d278a8f3e784de17
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1660325537221-62f68e83d278a8f3e784de17.jpeg
paper_authors.fullname: Ji Lin
paper_authors.type: user
paper_authors.user: jilinmit
paper_authors._id: 657920ef800326e825395dd2
paper_authors.name: Hongxu Yin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-30T07:25:33.390Z
paper_authors._id: 65a8b7f69aec1645994e7a15
paper_authors.avatarUrl: /avatars/debc086f3fea029db22847bde80799a0.svg
paper_authors.fullname: Hongxu Yin
paper_authors.type: user
paper_authors.user: yinhongxu
paper_authors._id: 657920ef800326e825395dd3
paper_authors.name: Wei Ping
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-15T08:00:55.606Z
paper_authors._id: 661cdb08ed360bdc4f6db37c
paper_authors.avatarUrl: /avatars/bced64e9b7f46125a03d6f6e66ae7e33.svg
paper_authors.fullname: Wei Ping
paper_authors.type: user
paper_authors.user: wpingnet
paper_authors._id: 657920ef800326e825395dd4
paper_authors.name: Yao Lu
paper_authors._id: 657920ef800326e825395dd5
paper_authors.name: Pavlo Molchanov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-20T20:45:43.355Z
paper_authors._id: 646d0c1c534e52f8c30500a6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
paper_authors.fullname: Pavlo Molchanov
paper_authors.type: user
paper_authors.user: pmolchanov
paper_authors._id: 657920ef800326e825395dd6
paper_authors.name: Andrew Tao
paper_authors._id: 657920ef800326e825395dd7
paper_authors.name: Huizi Mao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-13T11:02:46.583Z
paper_authors._id: 64bcd0468e051085baf09ec5
paper_authors.avatarUrl: /avatars/bd9ca61328d959c89d9c28384c17e712.svg
paper_authors.fullname: Huizi Mao
paper_authors.type: user
paper_authors.user: huizimaonv
paper_authors._id: 657920ef800326e825395dd8
paper_authors.name: Jan Kautz
paper_authors._id: 657920ef800326e825395dd9
paper_authors.name: Mohammad Shoeybi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-28T07:20:05.914Z
paper_authors._id: 6641544c695975af2cbd0da6
paper_authors.avatarUrl: /avatars/0ad3c18dcba585259b064fe9b00a07ce.svg
paper_authors.fullname: Mohammad Shoeybi
paper_authors.type: user
paper_authors.user: shoeybi
paper_authors._id: 657920ef800326e825395dda
paper_authors.name: Song Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-13T11:03:51.988Z
paper_authors._id: 63797f727df2fefdcaf3ff7e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1668906853549-noauth.jpeg
paper_authors.fullname: Song
paper_authors.type: user
paper_authors.user: songhan
paper_title: VILA: On Pre-training for Visual Language Models
paper_summary: Visual language models (VLMs) rapidly progressed with the recent success of
  large language models. There have been growing efforts on visual instruction
  tuning to extend the LLM with visual inputs, but lacks an in-depth study of the
  visual language pre-training process, where the model learns to perform joint
  modeling on both modalities. In this work, we examine the design options for
  VLM pre-training by augmenting LLM towards VLM through step-by-step
  controllable comparisons. We introduce three main findings: (1) freezing LLMs
  during pre-training can achieve decent zero-shot performance, but lack
  in-context learning capability, which requires unfreezing the LLM; (2)
  interleaved pre-training data is beneficial whereas image-text pairs alone are
  not optimal; (3) re-blending text-only instruction data to image-text data
  during instruction fine-tuning not only remedies the degradation of text-only
  tasks, but also boosts VLM task accuracy. With an enhanced pre-training recipe
  we build VILA, a Visual Language model family that consistently outperforms the
  state-of-the-art models, e.g., LLaVA-1.5, across main benchmarks without bells
  and whistles. Multi-modal pre-training also helps unveil appealing properties
  of VILA, including multi-image reasoning, enhanced in-context learning, and
  better world knowledge.
paper_ai_keywords: visual language models (VLMs) large language models (LLMs) visual inputs visual language pre-training process Joint modeling in-context learning capability freezing LLMs unfreezing LLM interleaved pre-training data image-text pairs text-only instruction data instruction fine-tuning VILA multi-image reasoning world knowledge