title: An Image is Worth 32 Tokens for Reconstruction and Generation
summary: Recent advancements in generative models have highlighted the crucial role of
  image tokenization in the efficient synthesis of high-resolution images.
  Tokenization, which transforms images into latent representations, reduces
  computational demands compared to directly processing pixels and enhances the
  effectiveness and efficiency of the generation process. Prior methods, such as
  VQGAN, typically utilize 2D latent grids with fixed downsampling factors.
  However, these 2D tokenizations face challenges in managing the inherent
  redundancies present in images, where adjacent regions frequently display
  similarities. To overcome this issue, we introduce Transformer-based
  1-Dimensional Tokenizer (TiTok), an innovative approach that tokenizes images
  into 1D latent sequences. TiTok provides a more compact latent representation,
  yielding substantially more efficient and effective representations than
  conventional techniques. For example, a 256 x 256 x 3 image can be reduced to
  just 32 discrete tokens, a significant reduction from the 256 or 1024 tokens
  obtained by prior methods. Despite its compact nature, TiTok achieves
  competitive performance to state-of-the-art approaches. Specifically, using the
  same generator framework, TiTok attains 1.97 gFID, outperforming MaskGIT
  baseline significantly by 4.21 at ImageNet 256 x 256 benchmark. The advantages
  of TiTok become even more significant when it comes to higher resolution. At
  ImageNet 512 x 512 benchmark, TiTok not only outperforms state-of-the-art
  diffusion model DiT-XL/2 (gFID 2.74 vs. 3.04), but also reduces the image
  tokens by 64x, leading to 410x faster generation process. Our best-performing
  variant can significantly surpasses DiT-XL/2 (gFID 2.13 vs. 3.04) while still
  generating high-quality samples 74x faster.
paper_authors._id: 666902d603e3c114f0651c3a
paper_authors.name: Qihang Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T07:52:29.465Z
paper_authors._id: 648010b00b9d0f49849adb19
paper_authors.avatarUrl: /avatars/54419bb6e6ee788aba10cd64cd921204.svg
paper_authors.fullname: Qihang Yu
paper_authors.type: user
paper_authors.user: yucornetto
paper_authors._id: 666902d603e3c114f0651c3b
paper_authors.name: Mark Weber
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-25T07:35:35.750Z
paper_authors._id: 651b3b452930812657c154ce
paper_authors.avatarUrl: /avatars/d424ef67b20e4277c4af44e05d1ea188.svg
paper_authors.fullname: Mark Weber
paper_authors.type: user
paper_authors.user: markweber
paper_authors._id: 666902d603e3c114f0651c3c
paper_authors.name: Xueqing Deng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T07:53:04.811Z
paper_authors._id: 65ca9b1743207e438a95e90c
paper_authors.avatarUrl: /avatars/8f7bde1c44d8e665a29ee08ce7fedfa4.svg
paper_authors.fullname: Xueqing Deng
paper_authors.type: user
paper_authors.user: xdeng77
paper_authors._id: 666902d603e3c114f0651c3d
paper_authors.name: Xiaohui Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-12T07:53:11.344Z
paper_authors._id: 6430aa1b32a732121cd81f98
paper_authors.avatarUrl: /avatars/5419f8d6d4d36fa5ac83e30667b9fd99.svg
paper_authors.fullname: Xiaohui Shen
paper_authors.type: user
paper_authors.user: XiaohuiShen
paper_authors._id: 666902d603e3c114f0651c3e
paper_authors.name: Daniel Cremers
paper_authors._id: 666902d603e3c114f0651c3f
paper_authors.name: Liang-Chieh Chen
paper_title: An Image is Worth 32 Tokens for Reconstruction and Generation
paper_summary: Recent advancements in generative models have highlighted the crucial role of
  image tokenization in the efficient synthesis of high-resolution images.
  Tokenization, which transforms images into latent representations, reduces
  computational demands compared to directly processing pixels and enhances the
  effectiveness and efficiency of the generation process. Prior methods, such as
  VQGAN, typically utilize 2D latent grids with fixed downsampling factors.
  However, these 2D tokenizations face challenges in managing the inherent
  redundancies present in images, where adjacent regions frequently display
  similarities. To overcome this issue, we introduce Transformer-based
  1-Dimensional Tokenizer (TiTok), an innovative approach that tokenizes images
  into 1D latent sequences. TiTok provides a more compact latent representation,
  yielding substantially more efficient and effective representations than
  conventional techniques. For example, a 256 x 256 x 3 image can be reduced to
  just 32 discrete tokens, a significant reduction from the 256 or 1024 tokens
  obtained by prior methods. Despite its compact nature, TiTok achieves
  competitive performance to state-of-the-art approaches. Specifically, using the
  same generator framework, TiTok attains 1.97 gFID, outperforming MaskGIT
  baseline significantly by 4.21 at ImageNet 256 x 256 benchmark. The advantages
  of TiTok become even more significant when it comes to higher resolution. At
  ImageNet 512 x 512 benchmark, TiTok not only outperforms state-of-the-art
  diffusion model DiT-XL/2 (gFID 2.74 vs. 3.04), but also reduces the image
  tokens by 64x, leading to 410x faster generation process. Our best-performing
  variant can significantly surpasses DiT-XL/2 (gFID 2.13 vs. 3.04) while still
  generating high-quality samples 74x faster.
paper_ai_keywords: image tokenization latent representations VQGAN 2D latent grids 2D tokenizations Transformer-based 1-Dimensional Tokenizer (TiTok) 1D latent sequences gFID MaskGIT DiT-XL/2 diffusion model high-resolution images