Autoregressive Semantic Visual Reconstruction Helps VLMs Understand
  Better
Typical large vision-language models (LVLMs) apply autoregressive supervision
solely to textual sequences, without fully incorporating the visual modality
into the learning process. This results in three key limitations: (1) an
inability to utilize images without accompanying captions, (2) the risk that
captions omit critical visual details, and (3) the challenge that certain
vision-centric content cannot be adequately conveyed through text. As a result,
current LVLMs often prioritize vision-to-language alignment while potentially
overlooking fine-grained visual information. While some prior works have
explored autoregressive image generation, effectively leveraging autoregressive
visual supervision to enhance image understanding remains an open challenge. In
this paper, we introduce Autoregressive Semantic Visual Reconstruction (ASVR),
which enables joint learning of visual and textual modalities within a unified
autoregressive framework. We show that autoregressively reconstructing the raw
visual appearance of images does not enhance and may even impair multimodal
understanding. In contrast, autoregressively reconstructing the semantic
representation of images consistently improves comprehension. Notably, we find
that even when models are given continuous image features as input, they can
effectively reconstruct discrete semantic tokens, resulting in stable and
consistent improvements across a wide range of multimodal understanding
benchmarks. Our approach delivers significant performance gains across varying
data scales (556k-2M) and types of LLM bacbones. Specifically, ASVR improves
LLaVA-1.5 by 5% in average scores across 14 multimodal benchmarks. The code is
available at https://github.com/AlenjandroWang/ASVR.
6848fff842e4f9106973f321
Dianyi Wang
6848fff842e4f9106973f322
Wei Song
claimed_verified
2025-06-11T20:21:19.066Z
665eccf5ffd59344a22533a8
/avatars/2ae2710753ce34a04937384bc6dddf70.svg
Wei Song
user
Songweii
6848fff842e4f9106973f323
Yikun Wang
claimed_verified
2025-06-11T16:02:20.996Z
627b73728b6ecd7ece822825
https://cdn-avatars.huggingface.co/v1/production/uploads/627b73728b6ecd7ece822825/QV-sT0vwupGZYg-loLPRw.jpeg
Yikun Wang
user
LibraTree
6848fff842e4f9106973f324
Siyuan Wang
6848fff842e4f9106973f325
Kaicheng Yu
6848fff842e4f9106973f326
Zhongyu Wei
6848fff842e4f9106973f327
Jiaqi Wang
claimed_verified
2025-06-18T12:17:20.045Z
66084749d81d611249a18239
/avatars/a7256b4e3b686133aaa481a72e3058de.svg
Alex Wang
user
Alex11556666
Autoregressive Semantic Visual Reconstruction Helps VLMs Understand
  Better
Typical large vision-language models (LVLMs) apply autoregressive supervision
solely to textual sequences, without fully incorporating the visual modality
into the learning process. This results in three key limitations: (1) an
inability to utilize images without accompanying captions, (2) the risk that
captions omit critical visual details, and (3) the challenge that certain
vision-centric content cannot be adequately conveyed through text. As a result,
current LVLMs often prioritize vision-to-language alignment while potentially
overlooking fine-grained visual information. While some prior works have
explored autoregressive image generation, effectively leveraging autoregressive
visual supervision to enhance image understanding remains an open challenge. In
this paper, we introduce Autoregressive Semantic Visual Reconstruction (ASVR),
which enables joint learning of visual and textual modalities within a unified
autoregressive framework. We show that autoregressively reconstructing the raw
visual appearance of images does not enhance and may even impair multimodal
understanding. In contrast, autoregressively reconstructing the semantic
representation of images consistently improves comprehension. Notably, we find
that even when models are given continuous image features as input, they can
effectively reconstruct discrete semantic tokens, resulting in stable and
consistent improvements across a wide range of multimodal understanding
benchmarks. Our approach delivers significant performance gains across varying
data scales (556k-2M) and types of LLM bacbones. Specifically, ASVR improves
LLaVA-1.5 by 5% in average scores across 14 multimodal benchmarks. The code is
available at https://github.com/AlenjandroWang/ASVR.
Autoregressive Semantic Visual Reconstruction (ASVR) improves multimodal understanding by focusing on semantic reconstruction rather than raw visual appearance, enhancing performance across various benchmarks.
autoregressive supervision large vision-language models (LVLMs) visual modality image captions autoregressive image generation multimodal learning semantic representation discrete semantic tokens multimodal understanding benchmarks LLaVA-1.5