Efficient Generative Modeling with Residual Vector Quantization-Based
  Tokens
We explore the use of Residual Vector Quantization (RVQ) for high-fidelity
generation in vector-quantized generative models. This quantization technique
maintains higher data fidelity by employing more in-depth tokens. However,
increasing the token number in generative models leads to slower inference
speeds. To this end, we introduce ResGen, an efficient RVQ-based discrete
diffusion model that generates high-fidelity samples without compromising
sampling speed. Our key idea is a direct prediction of vector embedding of
collective tokens rather than individual ones. Moreover, we demonstrate that
our proposed token masking and multi-token prediction method can be formulated
within a principled probabilistic framework using a discrete diffusion process
and variational inference. We validate the efficacy and generalizability of the
proposed method on two challenging tasks across different modalities:
conditional image generation} on ImageNet 256x256 and zero-shot text-to-speech
synthesis. Experimental results demonstrate that ResGen outperforms
autoregressive counterparts in both tasks, delivering superior performance
without compromising sampling speed. Furthermore, as we scale the depth of RVQ,
our generative models exhibit enhanced generation fidelity or faster sampling
speeds compared to similarly sized baseline models. The project page can be
found at https://resgen-genai.github.io
67603b58418a4b3626cf19c6
Jaehyeon Kim
claimed_verified
2024-12-17T08:04:10.474Z
64c890ebca915ffd302bade2
/avatars/9febfaee743b805fcfd3aada2e7dd64b.svg
Jaehyeon Kim
user
jaywalnut310
67603b58418a4b3626cf19c7
Taehong Moon
claimed_verified
2024-12-16T16:55:03.692Z
63de7eb015266dd945f470a8
/avatars/a1bc57c6a25b88a3b2848b9549895af2.svg
Taehong Moon
user
jason-moon
67603b58418a4b3626cf19c8
Keon Lee
67603b58418a4b3626cf19c9
Jaewoong Cho
Efficient Generative Modeling with Residual Vector Quantization-Based
  Tokens
We explore the use of Residual Vector Quantization (RVQ) for high-fidelity
generation in vector-quantized generative models. This quantization technique
maintains higher data fidelity by employing more in-depth tokens. However,
increasing the token number in generative models leads to slower inference
speeds. To this end, we introduce ResGen, an efficient RVQ-based discrete
diffusion model that generates high-fidelity samples without compromising
sampling speed. Our key idea is a direct prediction of vector embedding of
collective tokens rather than individual ones. Moreover, we demonstrate that
our proposed token masking and multi-token prediction method can be formulated
within a principled probabilistic framework using a discrete diffusion process
and variational inference. We validate the efficacy and generalizability of the
proposed method on two challenging tasks across different modalities:
conditional image generation} on ImageNet 256x256 and zero-shot text-to-speech
synthesis. Experimental results demonstrate that ResGen outperforms
autoregressive counterparts in both tasks, delivering superior performance
without compromising sampling speed. Furthermore, as we scale the depth of RVQ,
our generative models exhibit enhanced generation fidelity or faster sampling
speeds compared to similarly sized baseline models. The project page can be
found at https://resgen-genai.github.io
ResGen, an RVQ-based discrete diffusion model, achieves high-fidelity generation with faster inference speeds, outperforming autoregressive models in conditional image generation and zero-shot text-to-speech synthesis.
Residual Vector Quantization (RVQ) discrete diffusion model vector embedding token masking multi-token prediction probabilistic framework variational inference conditional image generation zero-shot text-to-speech synthesis