TokenFlow: Unified Image Tokenizer for Multimodal Understanding and
  Generation
We present TokenFlow, a novel unified image tokenizer that bridges the
long-standing gap between multimodal understanding and generation. Prior
research attempt to employ a single reconstruction-targeted Vector Quantization
(VQ) encoder for unifying these two tasks. We observe that understanding and
generation require fundamentally different granularities of visual information.
This leads to a critical trade-off, particularly compromising performance in
multimodal understanding tasks. TokenFlow addresses this challenge through an
innovative dual-codebook architecture that decouples semantic and pixel-level
feature learning while maintaining their alignment via a shared mapping
mechanism. This design enables direct access to both high-level semantic
representations crucial for understanding tasks and fine-grained visual
features essential for generation through shared indices. Our extensive
experiments demonstrate TokenFlow's superiority across multiple dimensions.
Leveraging TokenFlow, we demonstrate for the first time that discrete visual
input can surpass LLaVA-1.5 13B in understanding performance, achieving a 7.2\%
average improvement. For image reconstruction, we achieve a strong FID score of
0.63 at 384*384 resolution. Moreover, TokenFlow establishes state-of-the-art
performance in autoregressive image generation with a GenEval score of 0.55 at
256*256 resolution, achieving comparable results to SDXL.
6751173f4af3ce83f1c4304b
Liao Qu
claimed_verified
2024-12-05T09:18:17.156Z
64b796079ebb7e6c7ddcdabf
/avatars/51af43cab078705b8745b4f942f542e5.svg
Liao Qu
user
leo1117
6751173f4af3ce83f1c4304c
Huichao Zhang
admin_assigned
2024-12-05T10:30:59.799Z
636241c6b64a56694266509f
/avatars/90bf6156b649b78db13f08d0770951ad.svg
zhanghuichao
user
liqingzju
6751173f4af3ce83f1c4304d
Yiheng Liu
6751173f4af3ce83f1c4304e
Xu Wang
admin_assigned
2024-12-05T10:30:29.060Z
63058fb180bc5e03dad44526
/avatars/675eef729fa2b9305a19dc3d9dcbb6c7.svg
Xu Wang
user
xuwang
6751173f4af3ce83f1c4304f
Yi Jiang
claimed_verified
2025-03-02T20:19:22.720Z
6344dcb1cd37e44d9ed46508
https://cdn-avatars.huggingface.co/v1/production/uploads/6344dcb1cd37e44d9ed46508/J92UKSxKR3iziD2WJfih4.jpeg
Yi Jiang
user
JiangYi
6751173f4af3ce83f1c43050
Yiming Gao
admin_assigned
2024-12-05T10:30:21.929Z
64898022658cf96af857bbae
/avatars/6d6904ec9f1fee8b4b5d6009723cf271.svg
yiminggao
user
gaozong
6751173f4af3ce83f1c43051
Hu Ye
6751173f4af3ce83f1c43052
Daniel K. Du
6751173f4af3ce83f1c43053
Zehuan Yuan
admin_assigned
2024-12-05T10:29:52.065Z
661a80af3557013b638061d5
/avatars/4c551aeb223e257a5fc45b5b6c7ded49.svg
Zehuan Yuan
user
sweetrabor
6751173f4af3ce83f1c43054
Xinglong Wu
TokenFlow: Unified Image Tokenizer for Multimodal Understanding and
  Generation
We present TokenFlow, a novel unified image tokenizer that bridges the
long-standing gap between multimodal understanding and generation. Prior
research attempt to employ a single reconstruction-targeted Vector Quantization
(VQ) encoder for unifying these two tasks. We observe that understanding and
generation require fundamentally different granularities of visual information.
This leads to a critical trade-off, particularly compromising performance in
multimodal understanding tasks. TokenFlow addresses this challenge through an
innovative dual-codebook architecture that decouples semantic and pixel-level
feature learning while maintaining their alignment via a shared mapping
mechanism. This design enables direct access to both high-level semantic
representations crucial for understanding tasks and fine-grained visual
features essential for generation through shared indices. Our extensive
experiments demonstrate TokenFlow's superiority across multiple dimensions.
Leveraging TokenFlow, we demonstrate for the first time that discrete visual
input can surpass LLaVA-1.5 13B in understanding performance, achieving a 7.2\%
average improvement. For image reconstruction, we achieve a strong FID score of
0.63 at 384*384 resolution. Moreover, TokenFlow establishes state-of-the-art
performance in autoregressive image generation with a GenEval score of 0.55 at
256*256 resolution, achieving comparable results to SDXL.
TokenFlow, a dual-codebook image tokenizer, enhances multimodal understanding and generation by decoupling semantic and pixel-level features with shared indices.
Vector Quantization (VQ) dual-codebook architecture semantic representations pixel-level features FID score autoregressive image generation GenEval score LLaVA-1.5 SDXL