title: TokenFlow: Unified Image Tokenizer for Multimodal Understanding and
    Generation
summary: We present TokenFlow, a novel unified image tokenizer that bridges the
  long-standing gap between multimodal understanding and generation. Prior
  research attempt to employ a single reconstruction-targeted Vector Quantization
  (VQ) encoder for unifying these two tasks. We observe that understanding and
  generation require fundamentally different granularities of visual information.
  This leads to a critical trade-off, particularly compromising performance in
  multimodal understanding tasks. TokenFlow addresses this challenge through an
  innovative dual-codebook architecture that decouples semantic and pixel-level
  feature learning while maintaining their alignment via a shared mapping
  mechanism. This design enables direct access to both high-level semantic
  representations crucial for understanding tasks and fine-grained visual
  features essential for generation through shared indices. Our extensive
  experiments demonstrate TokenFlow's superiority across multiple dimensions.
  Leveraging TokenFlow, we demonstrate for the first time that discrete visual
  input can surpass LLaVA-1.5 13B in understanding performance, achieving a 7.2\%
  average improvement. For image reconstruction, we achieve a strong FID score of
  0.63 at 384*384 resolution. Moreover, TokenFlow establishes state-of-the-art
  performance in autoregressive image generation with a GenEval score of 0.55 at
  256*256 resolution, achieving comparable results to SDXL.
paper_authors._id: 6751173f4af3ce83f1c4304b
paper_authors.name: Liao Qu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-05T09:18:17.156Z
paper_authors._id: 64b796079ebb7e6c7ddcdabf
paper_authors.avatarUrl: /avatars/51af43cab078705b8745b4f942f542e5.svg
paper_authors.fullname: Liao Qu
paper_authors.type: user
paper_authors.user: leo1117
paper_authors._id: 6751173f4af3ce83f1c4304c
paper_authors.name: Huichao Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T10:30:59.799Z
paper_authors._id: 636241c6b64a56694266509f
paper_authors.avatarUrl: /avatars/90bf6156b649b78db13f08d0770951ad.svg
paper_authors.fullname: zhanghuichao
paper_authors.type: user
paper_authors.user: liqingzju
paper_authors._id: 6751173f4af3ce83f1c4304d
paper_authors.name: Yiheng Liu
paper_authors._id: 6751173f4af3ce83f1c4304e
paper_authors.name: Xu Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T10:30:29.060Z
paper_authors._id: 63058fb180bc5e03dad44526
paper_authors.avatarUrl: /avatars/1ce11be2c93a2165c4114ab1f45a2dd1.svg
paper_authors.fullname: Xu Wang
paper_authors.type: user
paper_authors.user: xuwang
paper_authors._id: 6751173f4af3ce83f1c4304f
paper_authors.name: Yi Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-02T20:19:22.720Z
paper_authors._id: 6344dcb1cd37e44d9ed46508
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6344dcb1cd37e44d9ed46508/J92UKSxKR3iziD2WJfih4.jpeg
paper_authors.fullname: Yi Jiang
paper_authors.type: user
paper_authors.user: JiangYi
paper_authors._id: 6751173f4af3ce83f1c43050
paper_authors.name: Yiming Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T10:30:21.929Z
paper_authors._id: 64898022658cf96af857bbae
paper_authors.avatarUrl: /avatars/6d6904ec9f1fee8b4b5d6009723cf271.svg
paper_authors.fullname: yiminggao
paper_authors.type: user
paper_authors.user: gaozong
paper_authors._id: 6751173f4af3ce83f1c43051
paper_authors.name: Hu Ye
paper_authors._id: 6751173f4af3ce83f1c43052
paper_authors.name: Daniel K. Du
paper_authors._id: 6751173f4af3ce83f1c43053
paper_authors.name: Zehuan Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-05T10:29:52.065Z
paper_authors._id: 661a80af3557013b638061d5
paper_authors.avatarUrl: /avatars/4c551aeb223e257a5fc45b5b6c7ded49.svg
paper_authors.fullname: Zehuan Yuan
paper_authors.type: user
paper_authors.user: sweetrabor
paper_authors._id: 6751173f4af3ce83f1c43054
paper_authors.name: Xinglong Wu
paper_title: TokenFlow: Unified Image Tokenizer for Multimodal Understanding and
    Generation
paper_summary: We present TokenFlow, a novel unified image tokenizer that bridges the
  long-standing gap between multimodal understanding and generation. Prior
  research attempt to employ a single reconstruction-targeted Vector Quantization
  (VQ) encoder for unifying these two tasks. We observe that understanding and
  generation require fundamentally different granularities of visual information.
  This leads to a critical trade-off, particularly compromising performance in
  multimodal understanding tasks. TokenFlow addresses this challenge through an
  innovative dual-codebook architecture that decouples semantic and pixel-level
  feature learning while maintaining their alignment via a shared mapping
  mechanism. This design enables direct access to both high-level semantic
  representations crucial for understanding tasks and fine-grained visual
  features essential for generation through shared indices. Our extensive
  experiments demonstrate TokenFlow's superiority across multiple dimensions.
  Leveraging TokenFlow, we demonstrate for the first time that discrete visual
  input can surpass LLaVA-1.5 13B in understanding performance, achieving a 7.2\%
  average improvement. For image reconstruction, we achieve a strong FID score of
  0.63 at 384*384 resolution. Moreover, TokenFlow establishes state-of-the-art
  performance in autoregressive image generation with a GenEval score of 0.55 at
  256*256 resolution, achieving comparable results to SDXL.
paper_ai_keywords: TokenFlow unified image tokenizer multimodal understanding generation Vector Quantization (VQ) dual-codebook architecture semantic feature learning pixel-level feature learning shared mapping mechanism high-level semantic representations fine-grained visual features discrete visual input LLaVA-1.5 13B understanding performance FID score autoregressive image generation GenEval score SDXL