title: EVEv2: Improved Baselines for Encoder-Free Vision-Language Models
summary: Existing encoder-free vision-language models (VLMs) are rapidly narrowing the
  performance gap with their encoder-based counterparts, highlighting the
  promising potential for unified multimodal systems with structural simplicity
  and efficient deployment. We systematically clarify the performance gap between
  VLMs using pre-trained vision encoders, discrete tokenizers, and minimalist
  visual layers from scratch, deeply excavating the under-examined
  characteristics of encoder-free VLMs. We develop efficient strategies for
  encoder-free VLMs that rival mainstream encoder-based ones. After an in-depth
  investigation, we launch EVEv2.0, a new and improved family of encoder-free
  VLMs. We show that: (i) Properly decomposing and hierarchically associating
  vision and language within a unified model reduces interference between
  modalities. (ii) A well-designed training strategy enables effective
  optimization for encoder-free VLMs. Through extensive evaluation, our EVEv2.0
  represents a thorough study for developing a decoder-only architecture across
  modalities, demonstrating superior data efficiency and strong vision-reasoning
  capability. Code is publicly available at: https://github.com/baaivision/EVE.
paper_authors._id: 67aac64de37429ebdbdafc40
paper_authors.name: Haiwen Diao
paper_authors._id: 67aac64de37429ebdbdafc41
paper_authors.name: Xiaotong Li
paper_authors._id: 67aac64de37429ebdbdafc42
paper_authors.name: Yufeng Cui
paper_authors._id: 67aac64de37429ebdbdafc43
paper_authors.name: Yueze Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T14:25:34.818Z
paper_authors._id: 6458b59c7a7e192202df8fa0
paper_authors.avatarUrl: /avatars/33ee716477e5686da8723d01e199cd27.svg
paper_authors.fullname: Yueze Wang
paper_authors.type: user
paper_authors.user: yzwang
paper_authors._id: 67aac64de37429ebdbdafc44
paper_authors.name: Haoge Deng
paper_authors._id: 67aac64de37429ebdbdafc45
paper_authors.name: Ting Pan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T07:55:09.401Z
paper_authors._id: 6565bc5ee5aac326bfc98e39
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vIfHy9Y1yAK6A96UCHNBH.jpeg
paper_authors.fullname: Ting Pan
paper_authors.type: user
paper_authors.user: PhyscalX
paper_authors._id: 67aac64de37429ebdbdafc46
paper_authors.name: Wenxuan Wang
paper_authors._id: 67aac64de37429ebdbdafc47
paper_authors.name: Huchuan Lu
paper_authors._id: 67aac64de37429ebdbdafc48
paper_authors.name: Xinlong Wang
paper_title: EVEv2: Improved Baselines for Encoder-Free Vision-Language Models
paper_summary: Existing encoder-free vision-language models (VLMs) are rapidly narrowing the
  performance gap with their encoder-based counterparts, highlighting the
  promising potential for unified multimodal systems with structural simplicity
  and efficient deployment. We systematically clarify the performance gap between
  VLMs using pre-trained vision encoders, discrete tokenizers, and minimalist
  visual layers from scratch, deeply excavating the under-examined
  characteristics of encoder-free VLMs. We develop efficient strategies for
  encoder-free VLMs that rival mainstream encoder-based ones. After an in-depth
  investigation, we launch EVEv2.0, a new and improved family of encoder-free
  VLMs. We show that: (i) Properly decomposing and hierarchically associating
  vision and language within a unified model reduces interference between
  modalities. (ii) A well-designed training strategy enables effective
  optimization for encoder-free VLMs. Through extensive evaluation, our EVEv2.0
  represents a thorough study for developing a decoder-only architecture across
  modalities, demonstrating superior data efficiency and strong vision-reasoning
  capability. Code is publicly available at: https://github.com/baaivision/EVE.
paper_ai_keywords: encoder-free vision-language models (VLMs) encoder-based counterparts unified multimodal systems visual layers discrete tokenizers inference-free VLMs decoder-only architecture data efficiency vision-reasoning capability