title: Finite Scalar Quantization: VQ-VAE Made Simple
summary: We propose to replace vector quantization (VQ) in the latent representation
  of VQ-VAEs with a simple scheme termed finite scalar quantization (FSQ), where
  we project the VAE representation down to a few dimensions (typically less than
  10). Each dimension is quantized to a small set of fixed values, leading to an
  (implicit) codebook given by the product of these sets. By appropriately
  choosing the number of dimensions and values each dimension can take, we obtain
  the same codebook size as in VQ. On top of such discrete representations, we
  can train the same models that have been trained on VQ-VAE representations. For
  example, autoregressive and masked transformer models for image generation,
  multimodal generation, and dense prediction computer vision tasks. Concretely,
  we employ FSQ with MaskGIT for image generation, and with UViM for depth
  estimation, colorization, and panoptic segmentation. Despite the much simpler
  design of FSQ, we obtain competitive performance in all these tasks. We
  emphasize that FSQ does not suffer from codebook collapse and does not need the
  complex machinery employed in VQ (commitment losses, codebook reseeding, code
  splitting, entropy penalties, etc.) to learn expressive discrete
  representations.
paper_authors._id: 6514fe93a18b3317c36bf154
paper_authors.name: Fabian Mentzer
paper_authors._id: 6514fe93a18b3317c36bf155
paper_authors.name: David Minnen
paper_authors._id: 6514fe93a18b3317c36bf156
paper_authors.name: Eirikur Agustsson
paper_authors._id: 6514fe93a18b3317c36bf157
paper_authors.name: Michael Tschannen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-28T09:08:41.046Z
paper_authors._id: 6489893e1ec8356ba5bb9777
paper_authors.avatarUrl: /avatars/54354c1e5774cadd1d83d42054e9d96b.svg
paper_authors.fullname: Michael Tschannen
paper_authors.type: user
paper_authors.user: mitsch
paper_title: Finite Scalar Quantization: VQ-VAE Made Simple
paper_summary: We propose to replace vector quantization (VQ) in the latent representation
  of VQ-VAEs with a simple scheme termed finite scalar quantization (FSQ), where
  we project the VAE representation down to a few dimensions (typically less than
  10). Each dimension is quantized to a small set of fixed values, leading to an
  (implicit) codebook given by the product of these sets. By appropriately
  choosing the number of dimensions and values each dimension can take, we obtain
  the same codebook size as in VQ. On top of such discrete representations, we
  can train the same models that have been trained on VQ-VAE representations. For
  example, autoregressive and masked transformer models for image generation,
  multimodal generation, and dense prediction computer vision tasks. Concretely,
  we employ FSQ with MaskGIT for image generation, and with UViM for depth
  estimation, colorization, and panoptic segmentation. Despite the much simpler
  design of FSQ, we obtain competitive performance in all these tasks. We
  emphasize that FSQ does not suffer from codebook collapse and does not need the
  complex machinery employed in VQ (commitment losses, codebook reseeding, code
  splitting, entropy penalties, etc.) to learn expressive discrete
  representations.
paper_ai_keywords: vector quantization (VQ) VQ-VAEs finite scalar quantization (FSQ) latent representation VAE representation quantized codebook categorical formulations batch and sequential Bayes updates autoregressive models stereo transformer models image generation multimodal generation dense prediction MaskGIT depth estimation colorization panoptic segmentation codebook collapse commitment losses codebook reseeding codebook splitting entropy penalties expressive discrete representations