title: ACDiT: Interpolating Autoregressive Conditional Modeling and Diffusion
    Transformer
summary: The recent surge of interest in comprehensive multimodal models has
  necessitated the unification of diverse modalities. However, the unification
  suffers from disparate methodologies. Continuous visual generation necessitates
  the full-sequence diffusion-based approach, despite its divergence from the
  autoregressive modeling in the text domain. We posit that autoregressive
  modeling, i.e., predicting the future based on past deterministic experience,
  remains crucial in developing both a visual generation model and a potential
  unified multimodal model. In this paper, we explore an interpolation between
  the autoregressive modeling and full-parameters diffusion to model visual
  information. At its core, we present ACDiT, an Autoregressive blockwise
  Conditional Diffusion Transformer, where the block size of diffusion, i.e., the
  size of autoregressive units, can be flexibly adjusted to interpolate between
  token-wise autoregression and full-sequence diffusion. ACDiT is easy to
  implement, as simple as creating a Skip-Causal Attention Mask (SCAM) during
  training. During inference, the process iterates between diffusion denoising
  and autoregressive decoding that can make full use of KV-Cache. We verify the
  effectiveness of ACDiT on image and video generation tasks. We also demonstrate
  that benefitted from autoregressive modeling, ACDiT can be seamlessly used in
  visual understanding tasks despite being trained on the diffusion objective.
  The analysis of the trade-off between autoregressive modeling and diffusion
  demonstrates the potential of ACDiT to be used in long-horizon visual
  generation tasks. These strengths make it promising as the backbone of future
  unified models.
paper_authors._id: 6759b7dbd98f3b3f28b5cad4
paper_authors.name: Jinyi Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-12T08:21:37.595Z
paper_authors._id: 637a06580a77f602dc4ac922
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637a06580a77f602dc4ac922/JR2VHWhG1tj3TKKNzD8GD.jpeg
paper_authors.fullname: Jinyi Hu
paper_authors.type: user
paper_authors.user: JamesHujy
paper_authors._id: 6759b7dbd98f3b3f28b5cad5
paper_authors.name: Shengding Hu
paper_authors._id: 6759b7dbd98f3b3f28b5cad6
paper_authors.name: Yuxuan Song
paper_authors._id: 6759b7dbd98f3b3f28b5cad7
paper_authors.name: Yufei Huang
paper_authors._id: 6759b7dbd98f3b3f28b5cad8
paper_authors.name: Mingxuan Wang
paper_authors._id: 6759b7dbd98f3b3f28b5cad9
paper_authors.name: Hao Zhou
paper_authors._id: 6759b7dbd98f3b3f28b5cada
paper_authors.name: Zhiyuan Liu
paper_authors._id: 6759b7dbd98f3b3f28b5cadb
paper_authors.name: Wei-Ying Ma
paper_authors._id: 6759b7dbd98f3b3f28b5cadc
paper_authors.name: Maosong Sun
paper_title: ACDiT: Interpolating Autoregressive Conditional Modeling and Diffusion
    Transformer
paper_summary: The recent surge of interest in comprehensive multimodal models has
  necessitated the unification of diverse modalities. However, the unification
  suffers from disparate methodologies. Continuous visual generation necessitates
  the full-sequence diffusion-based approach, despite its divergence from the
  autoregressive modeling in the text domain. We posit that autoregressive
  modeling, i.e., predicting the future based on past deterministic experience,
  remains crucial in developing both a visual generation model and a potential
  unified multimodal model. In this paper, we explore an interpolation between
  the autoregressive modeling and full-parameters diffusion to model visual
  information. At its core, we present ACDiT, an Autoregressive blockwise
  Conditional Diffusion Transformer, where the block size of diffusion, i.e., the
  size of autoregressive units, can be flexibly adjusted to interpolate between
  token-wise autoregression and full-sequence diffusion. ACDiT is easy to
  implement, as simple as creating a Skip-Causal Attention Mask (SCAM) during
  training. During inference, the process iterates between diffusion denoising
  and autoregressive decoding that can make full use of KV-Cache. We verify the
  effectiveness of ACDiT on image and video generation tasks. We also demonstrate
  that benefitted from autoregressive modeling, ACDiT can be seamlessly used in
  visual understanding tasks despite being trained on the diffusion objective.
  The analysis of the trade-off between autoregressive modeling and diffusion
  demonstrates the potential of ACDiT to be used in long-horizon visual
  generation tasks. These strengths make it promising as the backbone of future
  unified models.
paper_ai_keywords: diffusion-based approach autoregressive modeling multimodal models visual generation perceptual space pre-trained models ACDiT Autoregressive blockwise Conditional Diffusion Transformer Skip-Causal Attention Mask (SCAM) diffusion denoising autoregressive decoding KV-Cache long-horizon visual generation tasks