Autoregressive Model Beats Diffusion: Llama for Scalable Image
  Generation
We introduce LlamaGen, a new family of image generation models that apply
original ``next-token prediction'' paradigm of large language models to visual
generation domain. It is an affirmative answer to whether vanilla
autoregressive models, e.g., Llama, without inductive biases on visual signals
can achieve state-of-the-art image generation performance if scaling properly.
We reexamine design spaces of image tokenizers, scalability properties of image
generation models, and their training data quality. The outcome of this
exploration consists of: (1) An image tokenizer with downsample ratio of 16,
reconstruction quality of 0.94 rFID and codebook usage of 97% on ImageNet
benchmark. (2) A series of class-conditional image generation models ranging
from 111M to 3.1B parameters, achieving 2.18 FID on ImageNet 256x256
benchmarks, outperforming the popular diffusion models such as LDM, DiT. (3) A
text-conditional image generation model with 775M parameters, from two-stage
training on LAION-COCO and high aesthetics quality images, demonstrating
competitive performance of visual quality and text alignment. (4) We verify the
effectiveness of LLM serving frameworks in optimizing the inference speed of
image generation models and achieve 326% - 414% speedup. We release all models
and codes to facilitate open-source community of visual generation and
multimodal foundation models.
6667c71be490a870be48da80
Peize Sun
admin_assigned
2024-06-11T07:36:13.868Z
640dc9bf8512ec51d7f0ac1a
https://cdn-avatars.huggingface.co/v1/production/uploads/640dc9bf8512ec51d7f0ac1a/sT4rdEoQbzfW6D3xDVdqt.jpeg
peizesun
user
peizesun
6667c71be490a870be48da81
Yi Jiang
admin_assigned
2024-06-11T07:39:40.191Z
6344dcb1cd37e44d9ed46508
https://cdn-avatars.huggingface.co/v1/production/uploads/6344dcb1cd37e44d9ed46508/J92UKSxKR3iziD2WJfih4.jpeg
Yi Jiang
user
JiangYi
6667c71be490a870be48da82
Shoufa Chen
claimed_verified
2024-06-11T06:59:01.343Z
6412a33900634c4fe9873652
https://cdn-avatars.huggingface.co/v1/production/uploads/6412a33900634c4fe9873652/Nmn_yRA1gGD2VO1YbSOYF.jpeg
Shoufa Chen
user
ShoufaChen
6667c71be490a870be48da83
Shilong Zhang
admin_assigned
2024-06-11T07:40:05.731Z
6424ffce46d202ad3d918a67
https://cdn-avatars.huggingface.co/v1/production/uploads/6424ffce46d202ad3d918a67/gmYmOA072fP_5cJLc9Qs4.jpeg
Shilong Zhang
user
shilongz
6667c71be490a870be48da84
Bingyue Peng
6667c71be490a870be48da85
Ping Luo
6667c71be490a870be48da86
Zehuan Yuan
admin_assigned
2024-06-11T07:40:56.271Z
661a80af3557013b638061d5
/avatars/4c551aeb223e257a5fc45b5b6c7ded49.svg
Zehuan Yuan
user
sweetrabor
Autoregressive Model Beats Diffusion: Llama for Scalable Image
  Generation
We introduce LlamaGen, a new family of image generation models that apply
original ``next-token prediction'' paradigm of large language models to visual
generation domain. It is an affirmative answer to whether vanilla
autoregressive models, e.g., Llama, without inductive biases on visual signals
can achieve state-of-the-art image generation performance if scaling properly.
We reexamine design spaces of image tokenizers, scalability properties of image
generation models, and their training data quality. The outcome of this
exploration consists of: (1) An image tokenizer with downsample ratio of 16,
reconstruction quality of 0.94 rFID and codebook usage of 97% on ImageNet
benchmark. (2) A series of class-conditional image generation models ranging
from 111M to 3.1B parameters, achieving 2.18 FID on ImageNet 256x256
benchmarks, outperforming the popular diffusion models such as LDM, DiT. (3) A
text-conditional image generation model with 775M parameters, from two-stage
training on LAION-COCO and high aesthetics quality images, demonstrating
competitive performance of visual quality and text alignment. (4) We verify the
effectiveness of LLM serving frameworks in optimizing the inference speed of
image generation models and achieve 326% - 414% speedup. We release all models
and codes to facilitate open-source community of visual generation and
multimodal foundation models.
LlamaGen applies the next-token prediction paradigm from large language models to image generation, achieving state-of-the-art performance with various model sizes and conditions.
next-token prediction large language models autoregressive models image tokenizers class-conditional image generation text-conditional image generation FID rFID diffusion models LDM DiT multimodal foundation models