EVEv2: Improved Baselines for Encoder-Free Vision-Language Models
Existing encoder-free vision-language models (VLMs) are rapidly narrowing the
performance gap with their encoder-based counterparts, highlighting the
promising potential for unified multimodal systems with structural simplicity
and efficient deployment. We systematically clarify the performance gap between
VLMs using pre-trained vision encoders, discrete tokenizers, and minimalist
visual layers from scratch, deeply excavating the under-examined
characteristics of encoder-free VLMs. We develop efficient strategies for
encoder-free VLMs that rival mainstream encoder-based ones. After an in-depth
investigation, we launch EVEv2.0, a new and improved family of encoder-free
VLMs. We show that: (i) Properly decomposing and hierarchically associating
vision and language within a unified model reduces interference between
modalities. (ii) A well-designed training strategy enables effective
optimization for encoder-free VLMs. Through extensive evaluation, our EVEv2.0
represents a thorough study for developing a decoder-only architecture across
modalities, demonstrating superior data efficiency and strong vision-reasoning
capability. Code is publicly available at: https://github.com/baaivision/EVE.
67aac64de37429ebdbdafc40
Haiwen Diao
claimed_verified
2025-06-10T11:21:16.749Z
64b4a717aa03b6520839e9b8
https://cdn-avatars.huggingface.co/v1/production/uploads/64b4a717aa03b6520839e9b8/Rt3ERG-6BVEA4hAwOz0_I.jpeg
Haiwen Diao
user
Paranioar
67aac64de37429ebdbdafc41
Xiaotong Li
67aac64de37429ebdbdafc42
Yufeng Cui
67aac64de37429ebdbdafc43
Yueze Wang
claimed_verified
2025-02-11T14:25:34.818Z
6458b59c7a7e192202df8fa0
/avatars/33ee716477e5686da8723d01e199cd27.svg
Yueze Wang
user
yzwang
67aac64de37429ebdbdafc44
Haoge Deng
67aac64de37429ebdbdafc45
Ting Pan
claimed_verified
2025-02-11T07:55:09.401Z
6565bc5ee5aac326bfc98e39
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vIfHy9Y1yAK6A96UCHNBH.jpeg
Ting Pan
user
PhyscalX
67aac64de37429ebdbdafc46
Wenxuan Wang
67aac64de37429ebdbdafc47
Huchuan Lu
67aac64de37429ebdbdafc48
Xinlong Wang
EVEv2: Improved Baselines for Encoder-Free Vision-Language Models
Existing encoder-free vision-language models (VLMs) are rapidly narrowing the
performance gap with their encoder-based counterparts, highlighting the
promising potential for unified multimodal systems with structural simplicity
and efficient deployment. We systematically clarify the performance gap between
VLMs using pre-trained vision encoders, discrete tokenizers, and minimalist
visual layers from scratch, deeply excavating the under-examined
characteristics of encoder-free VLMs. We develop efficient strategies for
encoder-free VLMs that rival mainstream encoder-based ones. After an in-depth
investigation, we launch EVEv2.0, a new and improved family of encoder-free
VLMs. We show that: (i) Properly decomposing and hierarchically associating
vision and language within a unified model reduces interference between
modalities. (ii) A well-designed training strategy enables effective
optimization for encoder-free VLMs. Through extensive evaluation, our EVEv2.0
represents a thorough study for developing a decoder-only architecture across
modalities, demonstrating superior data efficiency and strong vision-reasoning
capability. Code is publicly available at: https://github.com/baaivision/EVE.
EVEv2.0 is a new family of encoder-free vision-language models that achieves superior performance through proper modality decomposition and effective training strategies.
encoder-free vision-language models VLMs pre-trained vision encoders discrete tokenizers minimalist visual layers unified model modality decomposition training strategy decoder-only architecture data efficiency vision-reasoning capability