Unveiling Encoder-Free Vision-Language Models
Existing vision-language models (VLMs) mostly rely on vision encoders to
extract visual features followed by large language models (LLMs) for
visual-language tasks. However, the vision encoders set a strong inductive bias
in abstracting visual representation, e.g., resolution, aspect ratio, and
semantic priors, which could impede the flexibility and efficiency of the VLMs.
Training pure VLMs that accept the seamless vision and language inputs, i.e.,
without vision encoders, remains challenging and rarely explored. Empirical
observations reveal that direct training without encoders results in slow
convergence and large performance gaps. In this work, we bridge the gap between
encoder-based and encoder-free models, and present a simple yet effective
training recipe towards pure VLMs. Specifically, we unveil the key aspects of
training encoder-free VLMs efficiently via thorough experiments: (1) Bridging
vision-language representation inside one unified decoder; (2) Enhancing visual
recognition capability via extra supervision. With these strategies, we launch
EVE, an encoder-free vision-language model that can be trained and forwarded
efficiently. Notably, solely utilizing 35M publicly accessible data, EVE can
impressively rival the encoder-based VLMs of similar capacities across multiple
vision-language benchmarks. It significantly outperforms the counterpart
Fuyu-8B with mysterious training procedures and undisclosed training data. We
believe that EVE provides a transparent and efficient route for developing a
pure decoder-only architecture across modalities. Our code and models are
publicly available at: https://github.com/baaivision/EVE.
66714eedcb8c849f4316e1c9
Haiwen Diao
claimed_verified
2024-07-08T08:51:10.327Z
64b4a717aa03b6520839e9b8
https://cdn-avatars.huggingface.co/v1/production/uploads/64b4a717aa03b6520839e9b8/Rt3ERG-6BVEA4hAwOz0_I.jpeg
Haiwen Diao
user
Paranioar
66714eedcb8c849f4316e1ca
Yufeng Cui
admin_assigned
2024-07-08T20:42:36.165Z
648683de623b5f050213f2be
/avatars/83ecbcf4a21f68d2893de79f0444d6e3.svg
Yufeng Cui
user
YufengCui
66714eedcb8c849f4316e1cb
Xiaotong Li
66714eedcb8c849f4316e1cc
Yueze Wang
claimed_verified
2024-09-27T07:00:01.955Z
6458b59c7a7e192202df8fa0
/avatars/33ee716477e5686da8723d01e199cd27.svg
Yueze Wang
user
yzwang
66714eedcb8c849f4316e1cd
Huchuan Lu
66714eedcb8c849f4316e1ce
Xinlong Wang
admin_assigned
2024-07-08T20:44:23.231Z
63ca558304c979828311c5a5
/avatars/2a439d79fba2f987cabe780d10c94d25.svg
Xinlong Wang
user
xinlongwang
Unveiling Encoder-Free Vision-Language Models
Existing vision-language models (VLMs) mostly rely on vision encoders to
extract visual features followed by large language models (LLMs) for
visual-language tasks. However, the vision encoders set a strong inductive bias
in abstracting visual representation, e.g., resolution, aspect ratio, and
semantic priors, which could impede the flexibility and efficiency of the VLMs.
Training pure VLMs that accept the seamless vision and language inputs, i.e.,
without vision encoders, remains challenging and rarely explored. Empirical
observations reveal that direct training without encoders results in slow
convergence and large performance gaps. In this work, we bridge the gap between
encoder-based and encoder-free models, and present a simple yet effective
training recipe towards pure VLMs. Specifically, we unveil the key aspects of
training encoder-free VLMs efficiently via thorough experiments: (1) Bridging
vision-language representation inside one unified decoder; (2) Enhancing visual
recognition capability via extra supervision. With these strategies, we launch
EVE, an encoder-free vision-language model that can be trained and forwarded
efficiently. Notably, solely utilizing 35M publicly accessible data, EVE can
impressively rival the encoder-based VLMs of similar capacities across multiple
vision-language benchmarks. It significantly outperforms the counterpart
Fuyu-8B with mysterious training procedures and undisclosed training data. We
believe that EVE provides a transparent and efficient route for developing a
pure decoder-only architecture across modalities. Our code and models are
publicly available at: https://github.com/baaivision/EVE.
EVE is an encoder-free vision-language model that achieves competitive performance on multiple benchmarks using a unified decoder and extra supervision.
vision-language models VLMs large language models LLMs vision encoders visual features visual representation abstract representation semantic priors convergence performance gaps encoder-free models unified decoder visual recognition capability extra supervision