Marrying Autoregressive Transformer and Diffusion with Multi-Reference
  Autoregression
We introduce TransDiff, the first image generation model that marries
Autoregressive (AR) Transformer with diffusion models. In this joint modeling
framework, TransDiff encodes labels and images into high-level semantic
features and employs a diffusion model to estimate the distribution of image
samples. On the ImageNet 256x256 benchmark, TransDiff significantly outperforms
other image generation models based on standalone AR Transformer or diffusion
models. Specifically, TransDiff achieves a Fr\'echet Inception Distance (FID)
of 1.61 and an Inception Score (IS) of 293.4, and further provides x2 faster
inference latency compared to state-of-the-art methods based on AR Transformer
and x112 faster inference compared to diffusion-only models. Furthermore,
building on the TransDiff model, we introduce a novel image generation paradigm
called Multi-Reference Autoregression (MRAR), which performs autoregressive
generation by predicting the next image. MRAR enables the model to reference
multiple previously generated images, thereby facilitating the learning of more
diverse representations and improving the quality of generated images in
subsequent iterations. By applying MRAR, the performance of TransDiff is
improved, with the FID reduced from 1.61 to 1.42. We expect TransDiff to open
up a new frontier in the field of image generation.
684eb86a60b4a34dbe007a0a
Dingcheng Zhen
claimed_verified
2025-06-17T13:06:49.402Z
6398433a1054c4954270182c
/avatars/5451d1d9bc79b6c6356c93fb82d4861c.svg
zhen dc
user
zhendch
684eb86a60b4a34dbe007a0b
Qian Qiao
claimed_verified
2025-06-23T08:30:47.795Z
656975b88931d28b2444163d
/avatars/7d94d5a76e56e719db69ac9d72217cd2.svg
joeqian
user
joeqianqian
684eb86a60b4a34dbe007a0c
Tan Yu
684eb86a60b4a34dbe007a0d
Kangxi Wu
684eb86a60b4a34dbe007a0e
Ziwei Zhang
684eb86a60b4a34dbe007a0f
Siyuan Liu
684eb86a60b4a34dbe007a10
Shunshun Yin
684eb86a60b4a34dbe007a11
Ming Tao
Marrying Autoregressive Transformer and Diffusion with Multi-Reference
  Autoregression
We introduce TransDiff, the first image generation model that marries
Autoregressive (AR) Transformer with diffusion models. In this joint modeling
framework, TransDiff encodes labels and images into high-level semantic
features and employs a diffusion model to estimate the distribution of image
samples. On the ImageNet 256x256 benchmark, TransDiff significantly outperforms
other image generation models based on standalone AR Transformer or diffusion
models. Specifically, TransDiff achieves a Fr\'echet Inception Distance (FID)
of 1.61 and an Inception Score (IS) of 293.4, and further provides x2 faster
inference latency compared to state-of-the-art methods based on AR Transformer
and x112 faster inference compared to diffusion-only models. Furthermore,
building on the TransDiff model, we introduce a novel image generation paradigm
called Multi-Reference Autoregression (MRAR), which performs autoregressive
generation by predicting the next image. MRAR enables the model to reference
multiple previously generated images, thereby facilitating the learning of more
diverse representations and improving the quality of generated images in
subsequent iterations. By applying MRAR, the performance of TransDiff is
improved, with the FID reduced from 1.61 to 1.42. We expect TransDiff to open
up a new frontier in the field of image generation.
TransDiff, combining an Autoregressive Transformer and diffusion models, achieves superior image generation performance and speed, while Multi-Reference Autoregression further enhances its quality and diversity.
Autoregressive (AR) Transformer diffusion models high-level semantic features Fr\'echet Inception Distance (FID) Inception Score (IS) Multi-Reference Autoregression (MRAR)