title: Improving Autoregressive Image Generation through Coarse-to-Fine Token
    Prediction
summary: Autoregressive models have shown remarkable success in image generation by
  adapting sequential prediction techniques from language modeling. However,
  applying these approaches to images requires discretizing continuous pixel data
  through vector quantization methods like VQ-VAE. To alleviate the quantization
  errors that existed in VQ-VAE, recent works tend to use larger codebooks.
  However, this will accordingly expand vocabulary size, complicating the
  autoregressive modeling task. This paper aims to find a way to enjoy the
  benefits of large codebooks without making autoregressive modeling more
  difficult. Through empirical investigation, we discover that tokens with
  similar codeword representations produce similar effects on the final generated
  image, revealing significant redundancy in large codebooks. Based on this
  insight, we propose to predict tokens from coarse to fine (CTF), realized by
  assigning the same coarse label for similar tokens. Our framework consists of
  two stages: (1) an autoregressive model that sequentially predicts coarse
  labels for each token in the sequence, and (2) an auxiliary model that
  simultaneously predicts fine-grained labels for all tokens conditioned on their
  coarse labels. Experiments on ImageNet demonstrate our method's superior
  performance, achieving an average improvement of 59 points in Inception Score
  compared to baselines. Notably, despite adding an inference step, our approach
  achieves faster sampling speeds.
paper_authors._id: 67dcf6375fd14aedd3005237
paper_authors.name: Ziyao Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T13:11:50.727Z
paper_authors._id: 64f1ba1cc640c6d957204786
paper_authors.avatarUrl: /avatars/28ede115da5334d2f22a4b5698df1c30.svg
paper_authors.fullname: Ziyao Guo
paper_authors.type: user
paper_authors.user: ZiyaoGuo
paper_authors._id: 67dcf6375fd14aedd3005238
paper_authors.name: Kaipeng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T13:12:11.686Z
paper_authors._id: 65f1713552c38a91e0a445e8
paper_authors.avatarUrl: /avatars/47ab3ada51c9b9976ac1cd0c4301c373.svg
paper_authors.fullname: kaipeng
paper_authors.type: user
paper_authors.user: kpzhang996
paper_authors._id: 67dcf6375fd14aedd3005239
paper_authors.name: Michael Qizhe Shieh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T13:12:23.239Z
paper_authors._id: 64d48e9ef2a3941669810f23
paper_authors.avatarUrl: /avatars/495e6785abc44d1facfcf385edd6a222.svg
paper_authors.fullname: Michael Shieh
paper_authors.type: user
paper_authors.user: Michael2008S
paper_title: Improving Autoregressive Image Generation through Coarse-to-Fine Token
    Prediction
paper_summary: Autoregressive models have shown remarkable success in image generation by
  adapting sequential prediction techniques from language modeling. However,
  applying these approaches to images requires discretizing continuous pixel data
  through vector quantization methods like VQ-VAE. To alleviate the quantization
  errors that existed in VQ-VAE, recent works tend to use larger codebooks.
  However, this will accordingly expand vocabulary size, complicating the
  autoregressive modeling task. This paper aims to find a way to enjoy the
  benefits of large codebooks without making autoregressive modeling more
  difficult. Through empirical investigation, we discover that tokens with
  similar codeword representations produce similar effects on the final generated
  image, revealing significant redundancy in large codebooks. Based on this
  insight, we propose to predict tokens from coarse to fine (CTF), realized by
  assigning the same coarse label for similar tokens. Our framework consists of
  two stages: (1) an autoregressive model that sequentially predicts coarse
  labels for each token in the sequence, and (2) an auxiliary model that
  simultaneously predicts fine-grained labels for all tokens conditioned on their
  coarse labels. Experiments on ImageNet demonstrate our method's superior
  performance, achieving an average improvement of 59 points in Inception Score
  compared to baselines. Notably, despite adding an inference step, our approach
  achieves faster sampling speeds.
paper_ai_summary: A coarse-to-fine approach improves image generation by optimizing autoregressive modeling with large codebooks, enhancing performance and speed.
paper_ai_keywords: autoregressive models image generation VQ-VAE vector quantization coarse-to-fine Inception Score