An Image is Worth 32 Tokens for Reconstruction and Generation
Recent advancements in generative models have highlighted the crucial role of
image tokenization in the efficient synthesis of high-resolution images.
Tokenization, which transforms images into latent representations, reduces
computational demands compared to directly processing pixels and enhances the
effectiveness and efficiency of the generation process. Prior methods, such as
VQGAN, typically utilize 2D latent grids with fixed downsampling factors.
However, these 2D tokenizations face challenges in managing the inherent
redundancies present in images, where adjacent regions frequently display
similarities. To overcome this issue, we introduce Transformer-based
1-Dimensional Tokenizer (TiTok), an innovative approach that tokenizes images
into 1D latent sequences. TiTok provides a more compact latent representation,
yielding substantially more efficient and effective representations than
conventional techniques. For example, a 256 x 256 x 3 image can be reduced to
just 32 discrete tokens, a significant reduction from the 256 or 1024 tokens
obtained by prior methods. Despite its compact nature, TiTok achieves
competitive performance to state-of-the-art approaches. Specifically, using the
same generator framework, TiTok attains 1.97 gFID, outperforming MaskGIT
baseline significantly by 4.21 at ImageNet 256 x 256 benchmark. The advantages
of TiTok become even more significant when it comes to higher resolution. At
ImageNet 512 x 512 benchmark, TiTok not only outperforms state-of-the-art
diffusion model DiT-XL/2 (gFID 2.74 vs. 3.04), but also reduces the image
tokens by 64x, leading to 410x faster generation process. Our best-performing
variant can significantly surpasses DiT-XL/2 (gFID 2.13 vs. 3.04) while still
generating high-quality samples 74x faster.
666902d603e3c114f0651c3a
Qihang Yu
admin_assigned
2024-06-12T07:52:29.465Z
648010b00b9d0f49849adb19
/avatars/54419bb6e6ee788aba10cd64cd921204.svg
Qihang Yu
user
yucornetto
666902d603e3c114f0651c3b
Mark Weber
claimed_verified
2024-09-25T07:35:35.750Z
651b3b452930812657c154ce
/avatars/d424ef67b20e4277c4af44e05d1ea188.svg
Mark Weber
user
markweber
666902d603e3c114f0651c3c
Xueqing Deng
admin_assigned
2024-06-12T07:53:04.811Z
65ca9b1743207e438a95e90c
/avatars/8f7bde1c44d8e665a29ee08ce7fedfa4.svg
Xueqing Deng
user
xdeng77
666902d603e3c114f0651c3d
Xiaohui Shen
admin_assigned
2024-06-12T07:53:11.344Z
6430aa1b32a732121cd81f98
/avatars/5419f8d6d4d36fa5ac83e30667b9fd99.svg
Xiaohui Shen
user
XiaohuiShen
666902d603e3c114f0651c3e
Daniel Cremers
666902d603e3c114f0651c3f
Liang-Chieh Chen
An Image is Worth 32 Tokens for Reconstruction and Generation
Recent advancements in generative models have highlighted the crucial role of
image tokenization in the efficient synthesis of high-resolution images.
Tokenization, which transforms images into latent representations, reduces
computational demands compared to directly processing pixels and enhances the
effectiveness and efficiency of the generation process. Prior methods, such as
VQGAN, typically utilize 2D latent grids with fixed downsampling factors.
However, these 2D tokenizations face challenges in managing the inherent
redundancies present in images, where adjacent regions frequently display
similarities. To overcome this issue, we introduce Transformer-based
1-Dimensional Tokenizer (TiTok), an innovative approach that tokenizes images
into 1D latent sequences. TiTok provides a more compact latent representation,
yielding substantially more efficient and effective representations than
conventional techniques. For example, a 256 x 256 x 3 image can be reduced to
just 32 discrete tokens, a significant reduction from the 256 or 1024 tokens
obtained by prior methods. Despite its compact nature, TiTok achieves
competitive performance to state-of-the-art approaches. Specifically, using the
same generator framework, TiTok attains 1.97 gFID, outperforming MaskGIT
baseline significantly by 4.21 at ImageNet 256 x 256 benchmark. The advantages
of TiTok become even more significant when it comes to higher resolution. At
ImageNet 512 x 512 benchmark, TiTok not only outperforms state-of-the-art
diffusion model DiT-XL/2 (gFID 2.74 vs. 3.04), but also reduces the image
tokens by 64x, leading to 410x faster generation process. Our best-performing
variant can significantly surpasses DiT-XL/2 (gFID 2.13 vs. 3.04) while still
generating high-quality samples 74x faster.
Transformer-based 1D Tokenizer (TiTok) reduces computational demands and enhances image synthesis efficiency by significantly decreasing token count and outperforming state-of-the-art methods in both speed and quality.
Tokenization VQGAN 1-dimensional Tokenizer Transformer-based latent representations gFID MaskGIT diffusion model DiT-XL/2