Factorized Visual Tokenization and Generation
Visual tokenizers are fundamental to image generation. They convert visual
data into discrete tokens, enabling transformer-based models to excel at image
generation. Despite their success, VQ-based tokenizers like VQGAN face
significant limitations due to constrained vocabulary sizes. Simply expanding
the codebook often leads to training instability and diminishing performance
gains, making scalability a critical challenge. In this work, we introduce
Factorized Quantization (FQ), a novel approach that revitalizes VQ-based
tokenizers by decomposing a large codebook into multiple independent
sub-codebooks. This factorization reduces the lookup complexity of large
codebooks, enabling more efficient and scalable visual tokenization. To ensure
each sub-codebook captures distinct and complementary information, we propose a
disentanglement regularization that explicitly reduces redundancy, promoting
diversity across the sub-codebooks. Furthermore, we integrate representation
learning into the training process, leveraging pretrained vision models like
CLIP and DINO to infuse semantic richness into the learned representations.
This design ensures our tokenizer captures diverse semantic levels, leading to
more expressive and disentangled representations. Experiments show that the
proposed FQGAN model substantially improves the reconstruction quality of
visual tokenizers, achieving state-of-the-art performance. We further
demonstrate that this tokenizer can be effectively adapted into auto-regressive
image generation. https://showlab.github.io/FQGAN
67455e2f6338ac803d291a30
Zechen Bai
admin_assigned
2024-11-26T22:48:42.619Z
64b7833aa5018e3c7c9b50d8
/avatars/782415605ed786b73f484fcc86a6384f.svg
Zechen Bai
user
ZechenBai
67455e2f6338ac803d291a31
Jianxiong Gao
claimed_verified
2024-11-26T21:20:57.000Z
643815c4961bb61e463c5896
/avatars/3b44592472f16c56105bff8c314d9939.svg
Jianxiong Gao
user
Jianxiong
67455e2f6338ac803d291a32
Ziteng Gao
admin_assigned
2024-11-26T22:48:47.633Z
648ac3d53470b17ccc90deaf
/avatars/35b678baa7a91959f40f704c5de2a3e1.svg
Ziteng Gao
user
sebgao
67455e2f6338ac803d291a33
Pichao Wang
admin_assigned
2024-11-26T22:48:55.654Z
65381380c104b8d4556301c3
/avatars/3b6e1283cba0e6a9147f7cfcebc01a0b.svg
Pichao Wang
user
wangpichao
67455e2f6338ac803d291a34
Zheng Zhang
admin_assigned
2024-11-26T22:49:02.397Z
648e9244e6598457da62a180
/avatars/60329251f08e93260196ff67a842c188.svg
Zheng Zhang
user
zhengzhang
67455e2f6338ac803d291a35
Tong He
admin_assigned
2024-11-26T22:49:08.894Z
64478c64e2148488340229db
/avatars/f5c23489a068e896381cdc25836ce3dd.svg
he
user
tonghe
67455e2f6338ac803d291a36
Mike Zheng Shou
admin_assigned
2024-11-26T22:49:15.050Z
661ab3da2b14565c7acccf5c
/avatars/fa4fc03664803e02aede4d4c3d50b393.svg
Mike Zheng Shou
user
AnalMom
Factorized Visual Tokenization and Generation
Visual tokenizers are fundamental to image generation. They convert visual
data into discrete tokens, enabling transformer-based models to excel at image
generation. Despite their success, VQ-based tokenizers like VQGAN face
significant limitations due to constrained vocabulary sizes. Simply expanding
the codebook often leads to training instability and diminishing performance
gains, making scalability a critical challenge. In this work, we introduce
Factorized Quantization (FQ), a novel approach that revitalizes VQ-based
tokenizers by decomposing a large codebook into multiple independent
sub-codebooks. This factorization reduces the lookup complexity of large
codebooks, enabling more efficient and scalable visual tokenization. To ensure
each sub-codebook captures distinct and complementary information, we propose a
disentanglement regularization that explicitly reduces redundancy, promoting
diversity across the sub-codebooks. Furthermore, we integrate representation
learning into the training process, leveraging pretrained vision models like
CLIP and DINO to infuse semantic richness into the learned representations.
This design ensures our tokenizer captures diverse semantic levels, leading to
more expressive and disentangled representations. Experiments show that the
proposed FQGAN model substantially improves the reconstruction quality of
visual tokenizers, achieving state-of-the-art performance. We further
demonstrate that this tokenizer can be effectively adapted into auto-regressive
image generation. https://showlab.github.io/FQGAN
Factorized Quantization (FQ) improves image generation by decomposing large codebooks into smaller, diverse sub-codebooks, enhancing scalability and reconstruction quality through disentanglement regularization and semantic representation learning.
Visual tokenizers transformer-based models VQGAN codebook lookup complexity factorization disentanglement regularization representation learning CLIP DINO FQGAN auto-regressive image generation