title: UniTok: A Unified Tokenizer for Visual Generation and Understanding
summary: The representation disparity between visual generation and understanding
  imposes a critical gap in integrating these capabilities into a single
  framework. To bridge this gap, we introduce UniTok, a discrete visual tokenizer
  that encodes fine-grained details for generation while also capturing
  high-level semantics for understanding. Despite recent studies have shown that
  these objectives could induce loss conflicts in training, we reveal that the
  underlying bottleneck stems from limited representational capacity of discrete
  tokens. We address this by introducing multi-codebook quantization, which
  divides vector quantization with several independent sub-codebooks to expand
  the latent feature space, while avoiding training instability caused by
  overlarge codebooks. Our method significantly raises the upper limit of unified
  discrete tokenizers to match or even surpass domain-specific continuous
  tokenizers. For instance, UniTok achieves a remarkable rFID of 0.38 (versus
  0.87 for SD-VAE) and a zero-shot accuracy of 78.6% (versus 76.2% for CLIP) on
  ImageNet. Our code is available at https://github.com/FoundationVision/UniTok.
paper_authors._id: 67c13c68d8247a49b808fdac
paper_authors.name: Chuofan Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:31:29.232Z
paper_authors._id: 62c585eb09baf76938a70de8
paper_authors.avatarUrl: /avatars/ae8cca53710b3325bf0dd0f08c2b1bbf.svg
paper_authors.fullname: Chuofan Ma
paper_authors.type: user
paper_authors.user: cfma
paper_authors._id: 67c13c68d8247a49b808fdad
paper_authors.name: Yi Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-02T20:18:07.566Z
paper_authors._id: 6344dcb1cd37e44d9ed46508
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6344dcb1cd37e44d9ed46508/J92UKSxKR3iziD2WJfih4.jpeg
paper_authors.fullname: Yi Jiang
paper_authors.type: user
paper_authors.user: JiangYi
paper_authors._id: 67c13c68d8247a49b808fdae
paper_authors.name: Junfeng Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:31:35.467Z
paper_authors._id: 6572ac949a5c2d6df9fab3c5
paper_authors.avatarUrl: /avatars/b5d70a86a452198381eee1c8f513ceec.svg
paper_authors.fullname: Junfeng Wu
paper_authors.type: user
paper_authors.user: JunfengWu
paper_authors._id: 67c13c68d8247a49b808fdaf
paper_authors.name: Jihan Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:31:41.522Z
paper_authors._id: 6304baf041387c7f1177a5d2
paper_authors.avatarUrl: /avatars/795c63f2394080eec78ca7981d4a1f78.svg
paper_authors.fullname: Jihan Yang
paper_authors.type: user
paper_authors.user: jihanyang
paper_authors._id: 67c13c68d8247a49b808fdb0
paper_authors.name: Xin Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-12T15:26:27.595Z
paper_authors._id: 62d1a71ac40bde8b986c3748
paper_authors.avatarUrl: /avatars/6dbfba55563c71ed0db0d59c2629d525.svg
paper_authors.fullname: Xin Yu
paper_authors.type: user
paper_authors.user: Andyx
paper_authors._id: 67c13c68d8247a49b808fdb1
paper_authors.name: Zehuan Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:32:06.178Z
paper_authors._id: 661a80af3557013b638061d5
paper_authors.avatarUrl: /avatars/4c551aeb223e257a5fc45b5b6c7ded49.svg
paper_authors.fullname: Zehuan Yuan
paper_authors.type: user
paper_authors.user: sweetrabor
paper_authors._id: 67c13c68d8247a49b808fdb2
paper_authors.name: Bingyue Peng
paper_authors._id: 67c13c68d8247a49b808fdb3
paper_authors.name: Xiaojuan Qi
paper_title: UniTok: A Unified Tokenizer for Visual Generation and Understanding
paper_summary: The representation disparity between visual generation and understanding
  imposes a critical gap in integrating these capabilities into a single
  framework. To bridge this gap, we introduce UniTok, a discrete visual tokenizer
  that encodes fine-grained details for generation while also capturing
  high-level semantics for understanding. Despite recent studies have shown that
  these objectives could induce loss conflicts in training, we reveal that the
  underlying bottleneck stems from limited representational capacity of discrete
  tokens. We address this by introducing multi-codebook quantization, which
  divides vector quantization with several independent sub-codebooks to expand
  the latent feature space, while avoiding training instability caused by
  overlarge codebooks. Our method significantly raises the upper limit of unified
  discrete tokenizers to match or even surpass domain-specific continuous
  tokenizers. For instance, UniTok achieves a remarkable rFID of 0.38 (versus
  0.87 for SD-VAE) and a zero-shot accuracy of 78.6% (versus 76.2% for CLIP) on
  ImageNet. Our code is available at https://github.com/FoundationVision/UniTok.
paper_ai_summary: UniTok, a discrete visual tokenizer utilizing multi-codebook quantization, achieves competitive performance in image generation and understanding tasks.
paper_ai_keywords: discrete visual tokenizer fine-grained details high-level semantics multi-codebook quantization vector quantization latent feature space representational capacity unified discrete tokenizers domain-specific continuous tokenizers rFID zero-shot accuracy ImageNet