Vision as a Dialect: Unifying Visual Understanding and Generation via
  Text-Aligned Representations
This paper presents a multimodal framework that attempts to unify visual
understanding and generation within a shared discrete semantic representation.
At its core is the Text-Aligned Tokenizer (TA-Tok), which converts images into
discrete tokens using a text-aligned codebook projected from a large language
model's (LLM) vocabulary. By integrating vision and text into a unified space
with an expanded vocabulary, our multimodal LLM, Tar, enables cross-modal input
and output through a shared interface, without the need for modality-specific
designs. Additionally, we propose scale-adaptive encoding and decoding to
balance efficiency and visual detail, along with a generative de-tokenizer to
produce high-fidelity visual outputs. To address diverse decoding needs, we
utilize two complementary de-tokenizers: a fast autoregressive model and a
diffusion-based model. To enhance modality fusion, we investigate advanced
pre-training tasks, demonstrating improvements in both visual understanding and
generation. Experiments across benchmarks show that Tar matches or surpasses
existing multimodal LLM methods, achieving faster convergence and greater
training efficiency. Code, models, and data are available at
https://tar.csuhan.com
685a07510e4ad7e2197584c6
Jiaming Han
claimed_verified
2025-06-24T08:08:47.088Z
62318c0386753f5f41d0e261
https://cdn-avatars.huggingface.co/v1/production/uploads/62318c0386753f5f41d0e261/xO_5PvOf7lXhQPnQLcmnq.jpeg
Jiaming Han
user
csuhan
685a07510e4ad7e2197584c7
Hao Chen
claimed_verified
2025-07-11T08:08:56.895Z
65171225a1a5e5d6177354e6
/avatars/4c4c8d0c511d4350463341b124aedb98.svg
hao chen
user
wanhu
685a07510e4ad7e2197584c8
Yang Zhao
685a07510e4ad7e2197584c9
Hanyu Wang
claimed_verified
2025-06-24T08:08:44.987Z
6365a174ad2c9e8b6731dd0f
/avatars/2f4dd0eda92bca5a7464129fe7d961f9.svg
Hanyu Wang
user
hywang66
685a07510e4ad7e2197584ca
Qi Zhao
685a07510e4ad7e2197584cb
Ziyan Yang
685a07510e4ad7e2197584cc
Hao He
685a07510e4ad7e2197584cd
Xiangyu Yue
admin_assigned
2025-06-24T12:50:57.558Z
666a8f24e2990b0cb16b7bf9
/avatars/fcbaf8f1e3e53a2a4a819b7cb2c53aa4.svg
Xiangyu Yue
user
xyyue
685a07510e4ad7e2197584ce
Lu Jiang
Vision as a Dialect: Unifying Visual Understanding and Generation via
  Text-Aligned Representations
This paper presents a multimodal framework that attempts to unify visual
understanding and generation within a shared discrete semantic representation.
At its core is the Text-Aligned Tokenizer (TA-Tok), which converts images into
discrete tokens using a text-aligned codebook projected from a large language
model's (LLM) vocabulary. By integrating vision and text into a unified space
with an expanded vocabulary, our multimodal LLM, Tar, enables cross-modal input
and output through a shared interface, without the need for modality-specific
designs. Additionally, we propose scale-adaptive encoding and decoding to
balance efficiency and visual detail, along with a generative de-tokenizer to
produce high-fidelity visual outputs. To address diverse decoding needs, we
utilize two complementary de-tokenizers: a fast autoregressive model and a
diffusion-based model. To enhance modality fusion, we investigate advanced
pre-training tasks, demonstrating improvements in both visual understanding and
generation. Experiments across benchmarks show that Tar matches or surpasses
existing multimodal LLM methods, achieving faster convergence and greater
training efficiency. Code, models, and data are available at
https://tar.csuhan.com
A multimodal framework uses a Text-Aligned Tokenizer (TA-Tok) to integrate vision and text into a unified space, employing a generative de-tokenizer with autoregressive and diffusion-based models for efficient and high-fidelity visual outputs.
Text-Aligned Tokenizer (TA-Tok) multimodal LLM Tar scale-adaptive encoding diffusion-based model autoregressive model modality fusion pre-training tasks