title: Distilled Decoding 1: One-step Sampling of Image Auto-regressive Models
    with Flow Matching
summary: Autoregressive (AR) models have achieved state-of-the-art performance in text
  and image generation but suffer from slow generation due to the token-by-token
  process. We ask an ambitious question: can a pre-trained AR model be adapted to
  generate outputs in just one or two steps? If successful, this would
  significantly advance the development and deployment of AR models. We notice
  that existing works that try to speed up AR generation by generating multiple
  tokens at once fundamentally cannot capture the output distribution due to the
  conditional dependencies between tokens, limiting their effectiveness for
  few-step generation. To address this, we propose Distilled Decoding (DD), which
  uses flow matching to create a deterministic mapping from Gaussian distribution
  to the output distribution of the pre-trained AR model. We then train a network
  to distill this mapping, enabling few-step generation. DD doesn't need the
  training data of the original AR model, making it more practical.We evaluate DD
  on state-of-the-art image AR models and present promising results on
  ImageNet-256. For VAR, which requires 10-step generation, DD enables one-step
  generation (6.3times speed-up), with an acceptable increase in FID from 4.19
  to 9.96. For LlamaGen, DD reduces generation from 256 steps to 1, achieving an
  217.8times speed-up with a comparable FID increase from 4.11 to 11.35. In
  both cases, baseline methods completely fail with FID>100. DD also excels on
  text-to-image generation, reducing the generation from 256 steps to 2 for
  LlamaGen with minimal FID increase from 25.70 to 28.95. As the first work to
  demonstrate the possibility of one-step generation for image AR models, DD
  challenges the prevailing notion that AR models are inherently slow, and opens
  up new opportunities for efficient AR generation. The project website is at
  https://imagination-research.github.io/distilled-decoding.
paper_authors._id: 676a3070b1618113354d99fa
paper_authors.name: Enshu Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:35:07.539Z
paper_authors._id: 63c7ecbcaa2a7669c45ad49e
paper_authors.avatarUrl: /avatars/eba0250d328b0d850d3a6d3057bea583.svg
paper_authors.fullname: Enshu Liu
paper_authors.type: user
paper_authors.user: jsttlgdkycy
paper_authors._id: 676a3070b1618113354d99fb
paper_authors.name: Xuefei Ning
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:35:05.401Z
paper_authors._id: 641031b1a78453b8d96b8420
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678782881444-noauth.jpeg
paper_authors.fullname: Xuefei Ning
paper_authors.type: user
paper_authors.user: Foxfi
paper_authors._id: 676a3070b1618113354d99fc
paper_authors.name: Yu Wang
paper_authors._id: 676a3070b1618113354d99fd
paper_authors.name: Zinan Lin
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-12-24T03:54:58.649Z
paper_authors._id: 64c832a8c547ed5243d29630
paper_authors.avatarUrl: /avatars/59d1975634e84095b69423c02441d453.svg
paper_authors.fullname: Zinan Lin
paper_authors.type: user
paper_authors.user: fjxmlzn
paper_title: Distilled Decoding 1: One-step Sampling of Image Auto-regressive Models
    with Flow Matching
paper_summary: Autoregressive (AR) models have achieved state-of-the-art performance in text
  and image generation but suffer from slow generation due to the token-by-token
  process. We ask an ambitious question: can a pre-trained AR model be adapted to
  generate outputs in just one or two steps? If successful, this would
  significantly advance the development and deployment of AR models. We notice
  that existing works that try to speed up AR generation by generating multiple
  tokens at once fundamentally cannot capture the output distribution due to the
  conditional dependencies between tokens, limiting their effectiveness for
  few-step generation. To address this, we propose Distilled Decoding (DD), which
  uses flow matching to create a deterministic mapping from Gaussian distribution
  to the output distribution of the pre-trained AR model. We then train a network
  to distill this mapping, enabling few-step generation. DD doesn't need the
  training data of the original AR model, making it more practical.We evaluate DD
  on state-of-the-art image AR models and present promising results on
  ImageNet-256. For VAR, which requires 10-step generation, DD enables one-step
  generation (6.3times speed-up), with an acceptable increase in FID from 4.19
  to 9.96. For LlamaGen, DD reduces generation from 256 steps to 1, achieving an
  217.8times speed-up with a comparable FID increase from 4.11 to 11.35. In
  both cases, baseline methods completely fail with FID>100. DD also excels on
  text-to-image generation, reducing the generation from 256 steps to 2 for
  LlamaGen with minimal FID increase from 25.70 to 28.95. As the first work to
  demonstrate the possibility of one-step generation for image AR models, DD
  challenges the prevailing notion that AR models are inherently slow, and opens
  up new opportunities for efficient AR generation. The project website is at
  https://imagination-research.github.io/distilled-decoding.
paper_ai_keywords: Autoregressive (AR) models token-by-token pre-trained AR model Distilled Decoding (DD) flow matching Gaussian distribution few-step generation ImageNet-256 VAR FID LlamaGen text-to-image generation