title: Efficient Generative Modeling with Residual Vector Quantization-Based
    Tokens
summary: We explore the use of Residual Vector Quantization (RVQ) for high-fidelity
  generation in vector-quantized generative models. This quantization technique
  maintains higher data fidelity by employing more in-depth tokens. However,
  increasing the token number in generative models leads to slower inference
  speeds. To this end, we introduce ResGen, an efficient RVQ-based discrete
  diffusion model that generates high-fidelity samples without compromising
  sampling speed. Our key idea is a direct prediction of vector embedding of
  collective tokens rather than individual ones. Moreover, we demonstrate that
  our proposed token masking and multi-token prediction method can be formulated
  within a principled probabilistic framework using a discrete diffusion process
  and variational inference. We validate the efficacy and generalizability of the
  proposed method on two challenging tasks across different modalities:
  conditional image generation} on ImageNet 256x256 and zero-shot text-to-speech
  synthesis. Experimental results demonstrate that ResGen outperforms
  autoregressive counterparts in both tasks, delivering superior performance
  without compromising sampling speed. Furthermore, as we scale the depth of RVQ,
  our generative models exhibit enhanced generation fidelity or faster sampling
  speeds compared to similarly sized baseline models. The project page can be
  found at https://resgen-genai.github.io
paper_authors._id: 67603b58418a4b3626cf19c6
paper_authors.name: Jaehyeon Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-17T08:04:10.474Z
paper_authors._id: 64c890ebca915ffd302bade2
paper_authors.avatarUrl: /avatars/9febfaee743b805fcfd3aada2e7dd64b.svg
paper_authors.fullname: Jaehyeon Kim
paper_authors.type: user
paper_authors.user: jaywalnut310
paper_authors._id: 67603b58418a4b3626cf19c7
paper_authors.name: Taehong Moon
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T16:55:03.692Z
paper_authors._id: 63de7eb015266dd945f470a8
paper_authors.avatarUrl: /avatars/a1bc57c6a25b88a3b2848b9549895af2.svg
paper_authors.fullname: Taehong Moon
paper_authors.type: user
paper_authors.user: jason-moon
paper_authors._id: 67603b58418a4b3626cf19c8
paper_authors.name: Keon Lee
paper_authors._id: 67603b58418a4b3626cf19c9
paper_authors.name: Jaewoong Cho
paper_title: Efficient Generative Modeling with Residual Vector Quantization-Based
    Tokens
paper_summary: We explore the use of Residual Vector Quantization (RVQ) for high-fidelity
  generation in vector-quantized generative models. This quantization technique
  maintains higher data fidelity by employing more in-depth tokens. However,
  increasing the token number in generative models leads to slower inference
  speeds. To this end, we introduce ResGen, an efficient RVQ-based discrete
  diffusion model that generates high-fidelity samples without compromising
  sampling speed. Our key idea is a direct prediction of vector embedding of
  collective tokens rather than individual ones. Moreover, we demonstrate that
  our proposed token masking and multi-token prediction method can be formulated
  within a principled probabilistic framework using a discrete diffusion process
  and variational inference. We validate the efficacy and generalizability of the
  proposed method on two challenging tasks across different modalities:
  conditional image generation} on ImageNet 256x256 and zero-shot text-to-speech
  synthesis. Experimental results demonstrate that ResGen outperforms
  autoregressive counterparts in both tasks, delivering superior performance
  without compromising sampling speed. Furthermore, as we scale the depth of RVQ,
  our generative models exhibit enhanced generation fidelity or faster sampling
  speeds compared to similarly sized baseline models. The project page can be
  found at https://resgen-genai.github.io
paper_ai_keywords: Residual Vector Quantization (RVQ) high-fidelity generation generative models discrete diffusion model vector embedding token masking multi-token prediction principled probabilistic framework discrete diffusion process variational inference conditional image generation zero-shot text-to-speech synthesis autoregressive counterparts enhanced generation fidelity