title: Unveiling Encoder-Free Vision-Language Models
summary: Existing vision-language models (VLMs) mostly rely on vision encoders to
  extract visual features followed by large language models (LLMs) for
  visual-language tasks. However, the vision encoders set a strong inductive bias
  in abstracting visual representation, e.g., resolution, aspect ratio, and
  semantic priors, which could impede the flexibility and efficiency of the VLMs.
  Training pure VLMs that accept the seamless vision and language inputs, i.e.,
  without vision encoders, remains challenging and rarely explored. Empirical
  observations reveal that direct training without encoders results in slow
  convergence and large performance gaps. In this work, we bridge the gap between
  encoder-based and encoder-free models, and present a simple yet effective
  training recipe towards pure VLMs. Specifically, we unveil the key aspects of
  training encoder-free VLMs efficiently via thorough experiments: (1) Bridging
  vision-language representation inside one unified decoder; (2) Enhancing visual
  recognition capability via extra supervision. With these strategies, we launch
  EVE, an encoder-free vision-language model that can be trained and forwarded
  efficiently. Notably, solely utilizing 35M publicly accessible data, EVE can
  impressively rival the encoder-based VLMs of similar capacities across multiple
  vision-language benchmarks. It significantly outperforms the counterpart
  Fuyu-8B with mysterious training procedures and undisclosed training data. We
  believe that EVE provides a transparent and efficient route for developing a
  pure decoder-only architecture across modalities. Our code and models are
  publicly available at: https://github.com/baaivision/EVE.
paper_authors._id: 66714eedcb8c849f4316e1c9
paper_authors.name: Haiwen Diao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-08T08:51:10.327Z
paper_authors._id: 64b4a717aa03b6520839e9b8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b4a717aa03b6520839e9b8/Rt3ERG-6BVEA4hAwOz0_I.jpeg
paper_authors.fullname: Haiwen Diao
paper_authors.type: user
paper_authors.user: Paranioar
paper_authors._id: 66714eedcb8c849f4316e1ca
paper_authors.name: Yufeng Cui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-08T20:42:36.165Z
paper_authors._id: 648683de623b5f050213f2be
paper_authors.avatarUrl: /avatars/83ecbcf4a21f68d2893de79f0444d6e3.svg
paper_authors.fullname: Yufeng Cui
paper_authors.type: user
paper_authors.user: YufengCui
paper_authors._id: 66714eedcb8c849f4316e1cb
paper_authors.name: Xiaotong Li
paper_authors._id: 66714eedcb8c849f4316e1cc
paper_authors.name: Yueze Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-27T07:00:01.955Z
paper_authors._id: 6458b59c7a7e192202df8fa0
paper_authors.avatarUrl: /avatars/33ee716477e5686da8723d01e199cd27.svg
paper_authors.fullname: Yueze Wang
paper_authors.type: user
paper_authors.user: yzwang
paper_authors._id: 66714eedcb8c849f4316e1cd
paper_authors.name: Huchuan Lu
paper_authors._id: 66714eedcb8c849f4316e1ce
paper_authors.name: Xinlong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-08T20:44:23.231Z
paper_authors._id: 63ca558304c979828311c5a5
paper_authors.avatarUrl: /avatars/2a439d79fba2f987cabe780d10c94d25.svg
paper_authors.fullname: Xinlong Wang
paper_authors.type: user
paper_authors.user: xinlongwang
paper_title: Unveiling Encoder-Free Vision-Language Models
paper_summary: Existing vision-language models (VLMs) mostly rely on vision encoders to
  extract visual features followed by large language models (LLMs) for
  visual-language tasks. However, the vision encoders set a strong inductive bias
  in abstracting visual representation, e.g., resolution, aspect ratio, and
  semantic priors, which could impede the flexibility and efficiency of the VLMs.
  Training pure VLMs that accept the seamless vision and language inputs, i.e.,
  without vision encoders, remains challenging and rarely explored. Empirical
  observations reveal that direct training without encoders results in slow
  convergence and large performance gaps. In this work, we bridge the gap between
  encoder-based and encoder-free models, and present a simple yet effective
  training recipe towards pure VLMs. Specifically, we unveil the key aspects of
  training encoder-free VLMs efficiently via thorough experiments: (1) Bridging
  vision-language representation inside one unified decoder; (2) Enhancing visual
  recognition capability via extra supervision. With these strategies, we launch
  EVE, an encoder-free vision-language model that can be trained and forwarded
  efficiently. Notably, solely utilizing 35M publicly accessible data, EVE can
  impressively rival the encoder-based VLMs of similar capacities across multiple
  vision-language benchmarks. It significantly outperforms the counterpart
  Fuyu-8B with mysterious training procedures and undisclosed training data. We
  believe that EVE provides a transparent and efficient route for developing a
  pure decoder-only architecture across modalities. Our code and models are
  publicly available at: https://github.com/baaivision/EVE.
paper_ai_keywords: encoder-free VLMs unified decoder vision-language representation visual recognition capability extra supervision vision-language benchmarks