title: Token-Shuffle: Towards High-Resolution Image Generation with
    Autoregressive Models
summary: Autoregressive (AR) models, long dominant in language generation, are
  increasingly applied to image synthesis but are often considered less
  competitive than Diffusion-based models. A primary limitation is the
  substantial number of image tokens required for AR models, which constrains
  both training and inference efficiency, as well as image resolution. To address
  this, we present Token-Shuffle, a novel yet simple method that reduces the
  number of image tokens in Transformer. Our key insight is the dimensional
  redundancy of visual vocabularies in Multimodal Large Language Models (MLLMs),
  where low-dimensional visual codes from visual encoder are directly mapped to
  high-dimensional language vocabularies. Leveraging this, we consider two key
  operations: token-shuffle, which merges spatially local tokens along channel
  dimension to decrease the input token number, and token-unshuffle, which
  untangles the inferred tokens after Transformer blocks to restore the spatial
  arrangement for output. Jointly training with textual prompts, our strategy
  requires no additional pretrained text-encoder and enables MLLMs to support
  extremely high-resolution image synthesis in a unified next-token prediction
  way while maintaining efficient training and inference. For the first time, we
  push the boundary of AR text-to-image generation to a resolution of 2048x2048
  with gratifying generation performance. In GenAI-benchmark, our 2.7B model
  achieves 0.77 overall score on hard prompts, outperforming AR models LlamaGen
  by 0.18 and diffusion models LDM by 0.15. Exhaustive large-scale human
  evaluations also demonstrate our prominent image generation ability in terms of
  text-alignment, visual flaw, and visual appearance. We hope that Token-Shuffle
  can serve as a foundational design for efficient high-resolution image
  generation within MLLMs.
paper_authors._id: 680b318bbbebf87944bc9595
paper_authors.name: Xu Ma
paper_authors._id: 680b318bbbebf87944bc9596
paper_authors.name: Peize Sun
paper_authors._id: 680b318bbbebf87944bc9597
paper_authors.name: Haoyu Ma
paper_authors._id: 680b318bbbebf87944bc9598
paper_authors.name: Hao Tang
paper_authors._id: 680b318bbbebf87944bc9599
paper_authors.name: Chih-Yao Ma
paper_authors._id: 680b318bbbebf87944bc959a
paper_authors.name: Jialiang Wang
paper_authors._id: 680b318bbbebf87944bc959b
paper_authors.name: Kunpeng Li
paper_authors._id: 680b318bbbebf87944bc959c
paper_authors.name: Xiaoliang Dai
paper_authors._id: 680b318bbbebf87944bc959d
paper_authors.name: Yujun Shi
paper_authors._id: 680b318bbbebf87944bc959e
paper_authors.name: Xuan Ju
paper_authors._id: 680b318bbbebf87944bc959f
paper_authors.name: Yushi Hu
paper_authors._id: 680b318bbbebf87944bc95a0
paper_authors.name: Artsiom Sanakoyeu
paper_authors._id: 680b318bbbebf87944bc95a1
paper_authors.name: Felix Juefei-Xu
paper_authors._id: 680b318bbbebf87944bc95a2
paper_authors.name: Ji Hou
paper_authors._id: 680b318bbbebf87944bc95a3
paper_authors.name: Junjiao Tian
paper_authors._id: 680b318bbbebf87944bc95a4
paper_authors.name: Tao Xu
paper_authors._id: 680b318bbbebf87944bc95a5
paper_authors.name: Tingbo Hou
paper_authors._id: 680b318bbbebf87944bc95a6
paper_authors.name: Yen-Cheng Liu
paper_authors._id: 680b318bbbebf87944bc95a7
paper_authors.name: Zecheng He
paper_authors._id: 680b318bbbebf87944bc95a8
paper_authors.name: Zijian He
paper_authors._id: 680b318bbbebf87944bc95a9
paper_authors.name: Matt Feiszli
paper_authors._id: 680b318bbbebf87944bc95aa
paper_authors.name: Peizhao Zhang
paper_authors._id: 680b318bbbebf87944bc95ab
paper_authors.name: Peter Vajda
paper_authors._id: 680b318bbbebf87944bc95ac
paper_authors.name: Sam Tsai
paper_authors._id: 680b318bbbebf87944bc95ad
paper_authors.name: Yun Fu
paper_title: Token-Shuffle: Towards High-Resolution Image Generation with
    Autoregressive Models
paper_summary: Autoregressive (AR) models, long dominant in language generation, are
  increasingly applied to image synthesis but are often considered less
  competitive than Diffusion-based models. A primary limitation is the
  substantial number of image tokens required for AR models, which constrains
  both training and inference efficiency, as well as image resolution. To address
  this, we present Token-Shuffle, a novel yet simple method that reduces the
  number of image tokens in Transformer. Our key insight is the dimensional
  redundancy of visual vocabularies in Multimodal Large Language Models (MLLMs),
  where low-dimensional visual codes from visual encoder are directly mapped to
  high-dimensional language vocabularies. Leveraging this, we consider two key
  operations: token-shuffle, which merges spatially local tokens along channel
  dimension to decrease the input token number, and token-unshuffle, which
  untangles the inferred tokens after Transformer blocks to restore the spatial
  arrangement for output. Jointly training with textual prompts, our strategy
  requires no additional pretrained text-encoder and enables MLLMs to support
  extremely high-resolution image synthesis in a unified next-token prediction
  way while maintaining efficient training and inference. For the first time, we
  push the boundary of AR text-to-image generation to a resolution of 2048x2048
  with gratifying generation performance. In GenAI-benchmark, our 2.7B model
  achieves 0.77 overall score on hard prompts, outperforming AR models LlamaGen
  by 0.18 and diffusion models LDM by 0.15. Exhaustive large-scale human
  evaluations also demonstrate our prominent image generation ability in terms of
  text-alignment, visual flaw, and visual appearance. We hope that Token-Shuffle
  can serve as a foundational design for efficient high-resolution image
  generation within MLLMs.
paper_ai_summary: A Token-Shuffle method enhances high-resolution AR image generation by reducing token numbers in Transformers, enabling efficient generation and outperforming diffusion models.
paper_ai_keywords: Autoregressive models Token-Shuffle token-shuffle token-unshuffle Multimodal Large Language Models visual vocabularies visual codes Transformer next-token prediction hard prompts GenAI-benchmark LlamaGen LDM text-alignment visual flaw visual appearance