title: Factorized Visual Tokenization and Generation
summary: Visual tokenizers are fundamental to image generation. They convert visual
  data into discrete tokens, enabling transformer-based models to excel at image
  generation. Despite their success, VQ-based tokenizers like VQGAN face
  significant limitations due to constrained vocabulary sizes. Simply expanding
  the codebook often leads to training instability and diminishing performance
  gains, making scalability a critical challenge. In this work, we introduce
  Factorized Quantization (FQ), a novel approach that revitalizes VQ-based
  tokenizers by decomposing a large codebook into multiple independent
  sub-codebooks. This factorization reduces the lookup complexity of large
  codebooks, enabling more efficient and scalable visual tokenization. To ensure
  each sub-codebook captures distinct and complementary information, we propose a
  disentanglement regularization that explicitly reduces redundancy, promoting
  diversity across the sub-codebooks. Furthermore, we integrate representation
  learning into the training process, leveraging pretrained vision models like
  CLIP and DINO to infuse semantic richness into the learned representations.
  This design ensures our tokenizer captures diverse semantic levels, leading to
  more expressive and disentangled representations. Experiments show that the
  proposed FQGAN model substantially improves the reconstruction quality of
  visual tokenizers, achieving state-of-the-art performance. We further
  demonstrate that this tokenizer can be effectively adapted into auto-regressive
  image generation. https://showlab.github.io/FQGAN
paper_authors._id: 67455e2f6338ac803d291a30
paper_authors.name: Zechen Bai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:48:42.619Z
paper_authors._id: 64b7833aa5018e3c7c9b50d8
paper_authors.avatarUrl: /avatars/782415605ed786b73f484fcc86a6384f.svg
paper_authors.fullname: Zechen Bai
paper_authors.type: user
paper_authors.user: ZechenBai
paper_authors._id: 67455e2f6338ac803d291a31
paper_authors.name: Jianxiong Gao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-26T21:20:57.000Z
paper_authors._id: 643815c4961bb61e463c5896
paper_authors.avatarUrl: /avatars/3b44592472f16c56105bff8c314d9939.svg
paper_authors.fullname: Jianxiong Gao
paper_authors.type: user
paper_authors.user: Jianxiong
paper_authors._id: 67455e2f6338ac803d291a32
paper_authors.name: Ziteng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:48:47.633Z
paper_authors._id: 648ac3d53470b17ccc90deaf
paper_authors.avatarUrl: /avatars/35b678baa7a91959f40f704c5de2a3e1.svg
paper_authors.fullname: Ziteng Gao
paper_authors.type: user
paper_authors.user: sebgao
paper_authors._id: 67455e2f6338ac803d291a33
paper_authors.name: Pichao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:48:55.654Z
paper_authors._id: 65381380c104b8d4556301c3
paper_authors.avatarUrl: /avatars/3b6e1283cba0e6a9147f7cfcebc01a0b.svg
paper_authors.fullname: Pichao Wang
paper_authors.type: user
paper_authors.user: wangpichao
paper_authors._id: 67455e2f6338ac803d291a34
paper_authors.name: Zheng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:49:02.397Z
paper_authors._id: 648e9244e6598457da62a180
paper_authors.avatarUrl: /avatars/60329251f08e93260196ff67a842c188.svg
paper_authors.fullname: Zheng Zhang
paper_authors.type: user
paper_authors.user: zhengzhang
paper_authors._id: 67455e2f6338ac803d291a35
paper_authors.name: Tong He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:49:08.894Z
paper_authors._id: 64478c64e2148488340229db
paper_authors.avatarUrl: /avatars/f5c23489a068e896381cdc25836ce3dd.svg
paper_authors.fullname: he
paper_authors.type: user
paper_authors.user: tonghe
paper_authors._id: 67455e2f6338ac803d291a36
paper_authors.name: Mike Zheng Shou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-26T22:49:15.050Z
paper_authors._id: 661ab3da2b14565c7acccf5c
paper_authors.avatarUrl: /avatars/fa4fc03664803e02aede4d4c3d50b393.svg
paper_authors.fullname: Mike Zheng Shou
paper_authors.type: user
paper_authors.user: AnalMom
paper_title: Factorized Visual Tokenization and Generation
paper_summary: Visual tokenizers are fundamental to image generation. They convert visual
  data into discrete tokens, enabling transformer-based models to excel at image
  generation. Despite their success, VQ-based tokenizers like VQGAN face
  significant limitations due to constrained vocabulary sizes. Simply expanding
  the codebook often leads to training instability and diminishing performance
  gains, making scalability a critical challenge. In this work, we introduce
  Factorized Quantization (FQ), a novel approach that revitalizes VQ-based
  tokenizers by decomposing a large codebook into multiple independent
  sub-codebooks. This factorization reduces the lookup complexity of large
  codebooks, enabling more efficient and scalable visual tokenization. To ensure
  each sub-codebook captures distinct and complementary information, we propose a
  disentanglement regularization that explicitly reduces redundancy, promoting
  diversity across the sub-codebooks. Furthermore, we integrate representation
  learning into the training process, leveraging pretrained vision models like
  CLIP and DINO to infuse semantic richness into the learned representations.
  This design ensures our tokenizer captures diverse semantic levels, leading to
  more expressive and disentangled representations. Experiments show that the
  proposed FQGAN model substantially improves the reconstruction quality of
  visual tokenizers, achieving state-of-the-art performance. We further
  demonstrate that this tokenizer can be effectively adapted into auto-regressive
  image generation. https://showlab.github.io/FQGAN
paper_ai_summary: Factorized Quantization (FQ) improves image generation by decomposing large codebooks into smaller, diverse sub-codebooks, enhancing scalability and reconstruction quality through disentanglement regularization and semantic representation learning.
paper_ai_keywords: Visual tokenizers transformer-based models VQGAN codebook lookup complexity factorization disentanglement regularization representation learning CLIP DINO FQGAN auto-regressive image generation