title: A Spark of Vision-Language Intelligence: 2-Dimensional Autoregressive
    Transformer for Efficient Finegrained Image Generation
summary: This work tackles the information loss bottleneck of vector-quantization (VQ)
  autoregressive image generation by introducing a novel model architecture
  called the 2-Dimensional Autoregression (DnD) Transformer. The DnD-Transformer
  predicts more codes for an image by introducing a new autoregression direction,
  model depth, along with the sequence length direction. Compared to
  traditional 1D autoregression and previous work utilizing similar 2D image
  decomposition such as RQ-Transformer, the DnD-Transformer is an end-to-end
  model that can generate higher quality images with the same backbone model size
  and sequence length, opening a new optimization perspective for autoregressive
  image generation. Furthermore, our experiments reveal that the
  DnD-Transformer's potential extends beyond generating natural images. It can
  even generate images with rich text and graphical elements in a self-supervised
  manner, demonstrating an understanding of these combined modalities. This has
  not been previously demonstrated for popular vision generative models such as
  diffusion models, showing a spark of vision-language intelligence when trained
  solely on images. Code, datasets and models are open at
  https://github.com/chenllliang/DnD-Transformer.
paper_authors._id: 67052598e2d3403eaf1fc3bc
paper_authors.name: Liang Chen
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-10-08T12:29:20.810Z
paper_authors._id: 61b0a4ce1b3d95b3d1ed9251
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Wwjr26vdudX5KYVTb8Q0a.png
paper_authors.fullname: Liang Chen
paper_authors.type: user
paper_authors.user: leonardPKU
paper_authors._id: 67052598e2d3403eaf1fc3bd
paper_authors.name: Sinan Tan
paper_authors._id: 67052598e2d3403eaf1fc3be
paper_authors.name: Zefan Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-09T11:21:21.768Z
paper_authors._id: 64b15284372d4340772a3dca
paper_authors.avatarUrl: /avatars/417d5f1bc1bcb5e4d5de6169673c2cf7.svg
paper_authors.fullname: Zefan Cai
paper_authors.type: user
paper_authors.user: ZefanCai
paper_authors._id: 67052598e2d3403eaf1fc3bf
paper_authors.name: Weichu Xie
paper_authors._id: 67052598e2d3403eaf1fc3c0
paper_authors.name: Haozhe Zhao
paper_authors._id: 67052598e2d3403eaf1fc3c1
paper_authors.name: Yichi Zhang
paper_authors._id: 67052598e2d3403eaf1fc3c2
paper_authors.name: Junyang Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-09T11:43:10.132Z
paper_authors._id: 620760a26e3b7210c2ff1943
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
paper_authors.fullname: Junyang Lin
paper_authors.type: user
paper_authors.user: JustinLin610
paper_authors._id: 67052598e2d3403eaf1fc3c3
paper_authors.name: Jinze Bai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-09T11:43:20.221Z
paper_authors._id: 60113fad51e116b62cd0a30e
paper_authors.avatarUrl: /avatars/469357d0a4a5d2e104ae5e32801b395d.svg
paper_authors.fullname: Jinze Bai
paper_authors.type: user
paper_authors.user: Jinze
paper_authors._id: 67052598e2d3403eaf1fc3c4
paper_authors.name: Tianyu Liu
paper_authors._id: 67052598e2d3403eaf1fc3c5
paper_authors.name: Baobao Chang
paper_title: A Spark of Vision-Language Intelligence: 2-Dimensional Autoregressive
    Transformer for Efficient Finegrained Image Generation
paper_summary: This work tackles the information loss bottleneck of vector-quantization (VQ)
  autoregressive image generation by introducing a novel model architecture
  called the 2-Dimensional Autoregression (DnD) Transformer. The DnD-Transformer
  predicts more codes for an image by introducing a new autoregression direction,
  model depth, along with the sequence length direction. Compared to
  traditional 1D autoregression and previous work utilizing similar 2D image
  decomposition such as RQ-Transformer, the DnD-Transformer is an end-to-end
  model that can generate higher quality images with the same backbone model size
  and sequence length, opening a new optimization perspective for autoregressive
  image generation. Furthermore, our experiments reveal that the
  DnD-Transformer's potential extends beyond generating natural images. It can
  even generate images with rich text and graphical elements in a self-supervised
  manner, demonstrating an understanding of these combined modalities. This has
  not been previously demonstrated for popular vision generative models such as
  diffusion models, showing a spark of vision-language intelligence when trained
  solely on images. Code, datasets and models are open at
  https://github.com/chenllliang/DnD-Transformer.
paper_ai_keywords: vector-quantization (VQ) autoregressive image generation 2-Dimensional Autoregression (DnD) Transformer model depth sequence length direction autoregression RQ-Transformer end-to-end model natural images rich text graphical elements self-supervised combined modalities diffusion models vision-language intelligence