title: Efficient-vDiT: Efficient Video Diffusion Transformers With Attention
    Tile
summary: Despite the promise of synthesizing high-fidelity videos, Diffusion
  Transformers (DiTs) with 3D full attention suffer from expensive inference due
  to the complexity of attention computation and numerous sampling steps. For
  example, the popular Open-Sora-Plan model consumes more than 9 minutes for
  generating a single video of 29 frames. This paper addresses the inefficiency
  issue from two aspects: 1) Prune the 3D full attention based on the redundancy
  within video data; We identify a prevalent tile-style repetitive pattern in the
  3D attention maps for video data, and advocate a new family of sparse 3D
  attention that holds a linear complexity w.r.t. the number of video frames. 2)
  Shorten the sampling process by adopting existing multi-step consistency
  distillation; We split the entire sampling trajectory into several segments and
  perform consistency distillation within each one to activate few-step
  generation capacities. We further devise a three-stage training pipeline to
  conjoin the low-complexity attention and few-step generation capacities.
  Notably, with 0.1% pretraining data, we turn the Open-Sora-Plan-1.2 model into
  an efficient one that is 7.4x -7.8x faster for 29 and 93 frames 720p video
  generation with a marginal performance trade-off in VBench. In addition, we
  demonstrate that our approach is amenable to distributed inference, achieving
  an additional 3.91x speedup when running on 4 GPUs with sequence parallelism.
paper_authors._id: 67aab9b4a2bf5e5ea03d4c19
paper_authors.name: Hangliang Ding
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T07:55:29.115Z
paper_authors._id: 643a451ee2b979ae6141329d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643a451ee2b979ae6141329d/HN3M5vyroanQoUEiXJFyB.jpeg
paper_authors.fullname: Hangliang Ding
paper_authors.type: user
paper_authors.user: foreverpiano
paper_authors._id: 67aab9b4a2bf5e5ea03d4c1a
paper_authors.name: Dacheng Li
paper_authors._id: 67aab9b4a2bf5e5ea03d4c1b
paper_authors.name: Runlong Su
paper_authors._id: 67aab9b4a2bf5e5ea03d4c1c
paper_authors.name: Peiyuan Zhang
paper_authors._id: 67aab9b4a2bf5e5ea03d4c1d
paper_authors.name: Zhijie Deng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T07:55:25.471Z
paper_authors._id: 64bba541da140e461924dfed
paper_authors.avatarUrl: /avatars/367993765b0ca3734b2b100db33ed787.svg
paper_authors.fullname: zhijie deng
paper_authors.type: user
paper_authors.user: zhijie3
paper_authors._id: 67aab9b4a2bf5e5ea03d4c1e
paper_authors.name: Ion Stoica
paper_authors._id: 67aab9b4a2bf5e5ea03d4c1f
paper_authors.name: Hao Zhang
paper_title: Efficient-vDiT: Efficient Video Diffusion Transformers With Attention
    Tile
paper_summary: Despite the promise of synthesizing high-fidelity videos, Diffusion
  Transformers (DiTs) with 3D full attention suffer from expensive inference due
  to the complexity of attention computation and numerous sampling steps. For
  example, the popular Open-Sora-Plan model consumes more than 9 minutes for
  generating a single video of 29 frames. This paper addresses the inefficiency
  issue from two aspects: 1) Prune the 3D full attention based on the redundancy
  within video data; We identify a prevalent tile-style repetitive pattern in the
  3D attention maps for video data, and advocate a new family of sparse 3D
  attention that holds a linear complexity w.r.t. the number of video frames. 2)
  Shorten the sampling process by adopting existing multi-step consistency
  distillation; We split the entire sampling trajectory into several segments and
  perform consistency distillation within each one to activate few-step
  generation capacities. We further devise a three-stage training pipeline to
  conjoin the low-complexity attention and few-step generation capacities.
  Notably, with 0.1% pretraining data, we turn the Open-Sora-Plan-1.2 model into
  an efficient one that is 7.4x -7.8x faster for 29 and 93 frames 720p video
  generation with a marginal performance trade-off in VBench. In addition, we
  demonstrate that our approach is amenable to distributed inference, achieving
  an additional 3.91x speedup when running on 4 GPUs with sequence parallelism.
paper_ai_summary: A method to improve the efficiency of video synthesis using Diffusion Transformers by pruning 3D full attention and shortening the sampling process, resulting in significant speedups with minimal performance loss.
paper_ai_keywords: Diffusion Transformers 3D full attention sparse 3D attention multi-step consistency distillation sequence parallelism