title: Accelerating Auto-regressive Text-to-Image Generation with Training-free
    Speculative Jacobi Decoding
summary: The current large auto-regressive models can generate high-quality,
  high-resolution images, but these models require hundreds or even thousands of
  steps of next-token prediction during inference, resulting in substantial time
  consumption. In existing studies, Jacobi decoding, an iterative parallel
  decoding algorithm, has been used to accelerate the auto-regressive generation
  and can be executed without training. However, the Jacobi decoding relies on a
  deterministic criterion to determine the convergence of iterations. Thus, it
  works for greedy decoding but is incompatible with sampling-based decoding
  which is crucial for visual quality and diversity in the current
  auto-regressive text-to-image generation. In this paper, we propose a
  training-free probabilistic parallel decoding algorithm, Speculative Jacobi
  Decoding (SJD), to accelerate auto-regressive text-to-image generation. By
  introducing a probabilistic convergence criterion, our SJD accelerates the
  inference of auto-regressive text-to-image generation while maintaining the
  randomness in sampling-based token decoding and allowing the model to generate
  diverse images. Specifically, SJD facilitates the model to predict multiple
  tokens at each step and accepts tokens based on the probabilistic criterion,
  enabling the model to generate images with fewer steps than the conventional
  next-token-prediction paradigm. We also investigate the token initialization
  strategies that leverage the spatial locality of visual data to further improve
  the acceleration ratio under specific scenarios. We conduct experiments for our
  proposed SJD on multiple auto-regressive text-to-image generation models,
  showing the effectiveness of model acceleration without sacrificing the visual
  quality.
paper_authors._id: 67020787400ad7197b4b4a5f
paper_authors.name: Yao Teng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-07T07:26:55.869Z
paper_authors._id: 6427e08288215cee63b1c44d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6427e08288215cee63b1c44d/rzaG978FF-ywzicWNl_xl.jpeg
paper_authors.fullname: yao teng
paper_authors.type: user
paper_authors.user: tytyt
paper_authors._id: 67020787400ad7197b4b4a60
paper_authors.name: Han Shi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-13T20:18:10.248Z
paper_authors._id: 637751ad7169ed981703d26a
paper_authors.avatarUrl: /avatars/9823f115040a3d6b0240038010e46cb6.svg
paper_authors.fullname: Han Shi
paper_authors.type: user
paper_authors.user: shihan96
paper_authors._id: 67020787400ad7197b4b4a61
paper_authors.name: Xian Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:55:47.337Z
paper_authors._id: 6423f10a3fa01ecba6fca861
paper_authors.avatarUrl: /avatars/e4b4c7dddc5cf962ade45d8bdfa10b1a.svg
paper_authors.fullname: Xian Liu
paper_authors.type: user
paper_authors.user: alvinliu0
paper_authors._id: 67020787400ad7197b4b4a62
paper_authors.name: Xuefei Ning
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:43:47.048Z
paper_authors._id: 641031b1a78453b8d96b8420
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678782881444-noauth.jpeg
paper_authors.fullname: Xuefei Ning
paper_authors.type: user
paper_authors.user: Foxfi
paper_authors._id: 67020787400ad7197b4b4a63
paper_authors.name: Guohao Dai
paper_authors._id: 67020787400ad7197b4b4a64
paper_authors.name: Yu Wang
paper_authors._id: 67020787400ad7197b4b4a65
paper_authors.name: Zhenguo Li
paper_authors._id: 67020787400ad7197b4b4a66
paper_authors.name: Xihui Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-16T09:44:29.460Z
paper_authors._id: 65d5ec74cd05bc1eaa125040
paper_authors.avatarUrl: /avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
paper_authors.fullname: Xihui Liu
paper_authors.type: user
paper_authors.user: XihuiLiu
paper_title: Accelerating Auto-regressive Text-to-Image Generation with Training-free
    Speculative Jacobi Decoding
paper_summary: The current large auto-regressive models can generate high-quality,
  high-resolution images, but these models require hundreds or even thousands of
  steps of next-token prediction during inference, resulting in substantial time
  consumption. In existing studies, Jacobi decoding, an iterative parallel
  decoding algorithm, has been used to accelerate the auto-regressive generation
  and can be executed without training. However, the Jacobi decoding relies on a
  deterministic criterion to determine the convergence of iterations. Thus, it
  works for greedy decoding but is incompatible with sampling-based decoding
  which is crucial for visual quality and diversity in the current
  auto-regressive text-to-image generation. In this paper, we propose a
  training-free probabilistic parallel decoding algorithm, Speculative Jacobi
  Decoding (SJD), to accelerate auto-regressive text-to-image generation. By
  introducing a probabilistic convergence criterion, our SJD accelerates the
  inference of auto-regressive text-to-image generation while maintaining the
  randomness in sampling-based token decoding and allowing the model to generate
  diverse images. Specifically, SJD facilitates the model to predict multiple
  tokens at each step and accepts tokens based on the probabilistic criterion,
  enabling the model to generate images with fewer steps than the conventional
  next-token-prediction paradigm. We also investigate the token initialization
  strategies that leverage the spatial locality of visual data to further improve
  the acceleration ratio under specific scenarios. We conduct experiments for our
  proposed SJD on multiple auto-regressive text-to-image generation models,
  showing the effectiveness of model acceleration without sacrificing the visual
  quality.
paper_ai_keywords: auto-regressive models next-token prediction Jacobi decoding parallel decoding algorithm greedy decoding sampling-based decoding speculative Jacobi Decoding probabilistic convergence criterion token decoding spatial locality