title: Causal Diffusion Transformers for Generative Modeling
summary: We introduce Causal Diffusion as the autoregressive (AR) counterpart of
  Diffusion models. It is a next-token(s) forecasting framework that is friendly
  to both discrete and continuous modalities and compatible with existing
  next-token prediction models like LLaMA and GPT. While recent works attempt to
  combine diffusion with AR models, we show that introducing sequential
  factorization to a diffusion model can substantially improve its performance
  and enables a smooth transition between AR and diffusion generation modes.
  Hence, we propose CausalFusion - a decoder-only transformer that
  dual-factorizes data across sequential tokens and diffusion noise levels,
  leading to state-of-the-art results on the ImageNet generation benchmark while
  also enjoying the AR advantage of generating an arbitrary number of tokens for
  in-context reasoning. We further demonstrate CausalFusion's multimodal
  capabilities through a joint image generation and captioning model, and
  showcase CausalFusion's ability for zero-shot in-context image manipulations.
  We hope that this work could provide the community with a fresh perspective on
  training multimodal models over discrete and continuous data.
paper_authors._id: 67610887df97bdafe87c37aa
paper_authors.name: Chaorui Deng
paper_authors._id: 67610887df97bdafe87c37ab
paper_authors.name: Deyao Zh
paper_authors._id: 67610887df97bdafe87c37ac
paper_authors.name: Kunchang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-17T08:03:39.007Z
paper_authors._id: 61fb81006374891646732f37
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1643872995181-61fb81006374891646732f37.jpeg
paper_authors.fullname: Kunchang Li
paper_authors.type: user
paper_authors.user: Andy1621
paper_authors._id: 67610887df97bdafe87c37ad
paper_authors.name: Shi Guan
paper_authors._id: 67610887df97bdafe87c37ae
paper_authors.name: Haoqi Fan
paper_title: Causal Diffusion Transformers for Generative Modeling
paper_summary: We introduce Causal Diffusion as the autoregressive (AR) counterpart of
  Diffusion models. It is a next-token(s) forecasting framework that is friendly
  to both discrete and continuous modalities and compatible with existing
  next-token prediction models like LLaMA and GPT. While recent works attempt to
  combine diffusion with AR models, we show that introducing sequential
  factorization to a diffusion model can substantially improve its performance
  and enables a smooth transition between AR and diffusion generation modes.
  Hence, we propose CausalFusion - a decoder-only transformer that
  dual-factorizes data across sequential tokens and diffusion noise levels,
  leading to state-of-the-art results on the ImageNet generation benchmark while
  also enjoying the AR advantage of generating an arbitrary number of tokens for
  in-context reasoning. We further demonstrate CausalFusion's multimodal
  capabilities through a joint image generation and captioning model, and
  showcase CausalFusion's ability for zero-shot in-context image manipulations.
  We hope that this work could provide the community with a fresh perspective on
  training multimodal models over discrete and continuous data.
paper_ai_keywords: Causal Diffusion autoregressive (AR) Diffusion models next-token(s) forecasting discrete modalities continuous modalities LLaMA GPT sequential factorization decoder-only transformer dual-factorization ImageNet generation benchmark in-context reasoning multimodal capabilities joint image generation captioning model zero-shot in-context image manipulations