VILA: On Pre-training for Visual Language Models
Visual language models (VLMs) rapidly progressed with the recent success of
large language models. There have been growing efforts on visual instruction
tuning to extend the LLM with visual inputs, but lacks an in-depth study of the
visual language pre-training process, where the model learns to perform joint
modeling on both modalities. In this work, we examine the design options for
VLM pre-training by augmenting LLM towards VLM through step-by-step
controllable comparisons. We introduce three main findings: (1) freezing LLMs
during pre-training can achieve decent zero-shot performance, but lack
in-context learning capability, which requires unfreezing the LLM; (2)
interleaved pre-training data is beneficial whereas image-text pairs alone are
not optimal; (3) re-blending text-only instruction data to image-text data
during instruction fine-tuning not only remedies the degradation of text-only
tasks, but also boosts VLM task accuracy. With an enhanced pre-training recipe
we build VILA, a Visual Language model family that consistently outperforms the
state-of-the-art models, e.g., LLaVA-1.5, across main benchmarks without bells
and whistles. Multi-modal pre-training also helps unveil appealing properties
of VILA, including multi-image reasoning, enhanced in-context learning, and
better world knowledge.
657920ef800326e825395dd1
Ji Lin
claimed_verified
2023-12-14T10:34:52.393Z
62f68e83d278a8f3e784de17
https://cdn-avatars.huggingface.co/v1/production/uploads/1660325537221-62f68e83d278a8f3e784de17.jpeg
Ji Lin
user
jilinmit
657920ef800326e825395dd2
Hongxu Yin
claimed_verified
2024-05-30T07:25:33.390Z
65a8b7f69aec1645994e7a15
/avatars/debc086f3fea029db22847bde80799a0.svg
Hongxu Yin
user
yinhongxu
657920ef800326e825395dd3
Wei Ping
claimed_verified
2024-04-15T08:00:55.606Z
661cdb08ed360bdc4f6db37c
/avatars/bced64e9b7f46125a03d6f6e66ae7e33.svg
Wei Ping
user
wpingnet
657920ef800326e825395dd4
Yao Lu
657920ef800326e825395dd5
Pavlo Molchanov
claimed_verified
2023-12-20T20:45:43.355Z
646d0c1c534e52f8c30500a6
https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
Pavlo Molchanov
user
pmolchanov
657920ef800326e825395dd6
Andrew Tao
657920ef800326e825395dd7
Huizi Mao
admin_assigned
2023-12-13T11:02:46.583Z
64bcd0468e051085baf09ec5
/avatars/bd9ca61328d959c89d9c28384c17e712.svg
Huizi Mao
user
huizimaonv
657920ef800326e825395dd8
Jan Kautz
657920ef800326e825395dd9
Mohammad Shoeybi
claimed_verified
2024-05-28T07:20:05.914Z
6641544c695975af2cbd0da6
/avatars/0ad3c18dcba585259b064fe9b00a07ce.svg
Mohammad Shoeybi
user
shoeybi
657920ef800326e825395dda
Song Han
admin_assigned
2023-12-13T11:03:51.988Z
63797f727df2fefdcaf3ff7e
https://cdn-avatars.huggingface.co/v1/production/uploads/1668906853549-noauth.jpeg
Song
user
songhan
VILA: On Pre-training for Visual Language Models
Visual language models (VLMs) rapidly progressed with the recent success of
large language models. There have been growing efforts on visual instruction
tuning to extend the LLM with visual inputs, but lacks an in-depth study of the
visual language pre-training process, where the model learns to perform joint
modeling on both modalities. In this work, we examine the design options for
VLM pre-training by augmenting LLM towards VLM through step-by-step
controllable comparisons. We introduce three main findings: (1) freezing LLMs
during pre-training can achieve decent zero-shot performance, but lack
in-context learning capability, which requires unfreezing the LLM; (2)
interleaved pre-training data is beneficial whereas image-text pairs alone are
not optimal; (3) re-blending text-only instruction data to image-text data
during instruction fine-tuning not only remedies the degradation of text-only
tasks, but also boosts VLM task accuracy. With an enhanced pre-training recipe
we build VILA, a Visual Language model family that consistently outperforms the
state-of-the-art models, e.g., LLaVA-1.5, across main benchmarks without bells
and whistles. Multi-modal pre-training also helps unveil appealing properties
of VILA, including multi-image reasoning, enhanced in-context learning, and
better world knowledge.
Enhanced pre-training methods improve visual language models by balancing LLM freezing, interleaved data use, and text-only instruction re-blending, leading to superior performance across benchmarks.
visual language models large language models visual instruction tuning zero-shot performance in-context learning interleaved pre-training image-text pairs text-only instruction data multi-modal pre-training multi-image reasoning world knowledge