Distilled Decoding 1: One-step Sampling of Image Auto-regressive Models
  with Flow Matching
Autoregressive (AR) models have achieved state-of-the-art performance in text
and image generation but suffer from slow generation due to the token-by-token
process. We ask an ambitious question: can a pre-trained AR model be adapted to
generate outputs in just one or two steps? If successful, this would
significantly advance the development and deployment of AR models. We notice
that existing works that try to speed up AR generation by generating multiple
tokens at once fundamentally cannot capture the output distribution due to the
conditional dependencies between tokens, limiting their effectiveness for
few-step generation. To address this, we propose Distilled Decoding (DD), which
uses flow matching to create a deterministic mapping from Gaussian distribution
to the output distribution of the pre-trained AR model. We then train a network
to distill this mapping, enabling few-step generation. DD doesn't need the
training data of the original AR model, making it more practical.We evaluate DD
on state-of-the-art image AR models and present promising results on
ImageNet-256. For VAR, which requires 10-step generation, DD enables one-step
generation (6.3times speed-up), with an acceptable increase in FID from 4.19
to 9.96. For LlamaGen, DD reduces generation from 256 steps to 1, achieving an
217.8times speed-up with a comparable FID increase from 4.11 to 11.35. In
both cases, baseline methods completely fail with FID>100. DD also excels on
text-to-image generation, reducing the generation from 256 steps to 2 for
LlamaGen with minimal FID increase from 25.70 to 28.95. As the first work to
demonstrate the possibility of one-step generation for image AR models, DD
challenges the prevailing notion that AR models are inherently slow, and opens
up new opportunities for efficient AR generation. The project website is at
https://imagination-research.github.io/distilled-decoding.
676a3070b1618113354d99fa
Enshu Liu
claimed_verified
2024-12-30T19:35:07.539Z
63c7ecbcaa2a7669c45ad49e
/avatars/eba0250d328b0d850d3a6d3057bea583.svg
Enshu Liu
user
jsttlgdkycy
676a3070b1618113354d99fb
Xuefei Ning
claimed_verified
2024-12-30T19:35:05.401Z
641031b1a78453b8d96b8420
https://cdn-avatars.huggingface.co/v1/production/uploads/1678782881444-noauth.jpeg
Xuefei Ning
user
Foxfi
676a3070b1618113354d99fc
Yu Wang
676a3070b1618113354d99fd
Zinan Lin
extracted_confirmed
2024-12-24T03:54:58.649Z
64c832a8c547ed5243d29630
https://cdn-avatars.huggingface.co/v1/production/uploads/64c832a8c547ed5243d29630/a46V0xOVRVknM9mUjKkTf.jpeg
Zinan Lin
user
fjxmlzn
Distilled Decoding 1: One-step Sampling of Image Auto-regressive Models
  with Flow Matching
Autoregressive (AR) models have achieved state-of-the-art performance in text
and image generation but suffer from slow generation due to the token-by-token
process. We ask an ambitious question: can a pre-trained AR model be adapted to
generate outputs in just one or two steps? If successful, this would
significantly advance the development and deployment of AR models. We notice
that existing works that try to speed up AR generation by generating multiple
tokens at once fundamentally cannot capture the output distribution due to the
conditional dependencies between tokens, limiting their effectiveness for
few-step generation. To address this, we propose Distilled Decoding (DD), which
uses flow matching to create a deterministic mapping from Gaussian distribution
to the output distribution of the pre-trained AR model. We then train a network
to distill this mapping, enabling few-step generation. DD doesn't need the
training data of the original AR model, making it more practical.We evaluate DD
on state-of-the-art image AR models and present promising results on
ImageNet-256. For VAR, which requires 10-step generation, DD enables one-step
generation (6.3times speed-up), with an acceptable increase in FID from 4.19
to 9.96. For LlamaGen, DD reduces generation from 256 steps to 1, achieving an
217.8times speed-up with a comparable FID increase from 4.11 to 11.35. In
both cases, baseline methods completely fail with FID>100. DD also excels on
text-to-image generation, reducing the generation from 256 steps to 2 for
LlamaGen with minimal FID increase from 25.70 to 28.95. As the first work to
demonstrate the possibility of one-step generation for image AR models, DD
challenges the prevailing notion that AR models are inherently slow, and opens
up new opportunities for efficient AR generation. The project website is at
https://imagination-research.github.io/distilled-decoding.
Distilled Decoding accelerates autoregressive model generation by enabling one-step or two-step output, with improved efficiency and acceptable quality, challenging the notion that AR models are necessarily slow.
autoregressive models token-by-token process distilled decoding flow matching Gaussian distribution ImageNet-256 VAR LlamaGen text-to-image generation FID