title: Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate
    Video Diffusion Transformers
summary: While Diffusion Transformers (DiTs) have achieved breakthroughs in video
  generation, this long sequence generation task remains constrained by the
  quadratic complexity of attention mechanisms, resulting in significant
  inference latency. Through detailed analysis of attention maps in Video
  Diffusion Transformer (vDiT), we identify three recurring sparsity patterns:
  diagonal, multi-diagonal, and vertical-stripe structures. And even 3-6\%
  attention heads can be skipped. Crucially, these patterns exhibit strong
  layer-depth and head-position correlations but show limited dependence on the
  input content. Leveraging these findings, we propose Sparse-vDiT, a sparsity
  acceleration framework for vDiT comprising: 1) Pattern-optimized sparse kernels
  that replace dense attention with computationally efficient implementations for
  each identified sparsity pattern. 2) An offline sparse diffusion search
  algorithm that selects the optimal sparse computation strategy per layer and
  head via hardware-aware cost modeling. After determining the optimal
  configuration, we fuse heads within the same layer that share the same
  attention strategy, enhancing inference efficiency. Integrated into
  state-of-the-art vDiT models (CogVideoX1.5, HunyuanVideo, and Wan2.1),
  Sparse-vDiT achieves 2.09times, 2.38times, and 1.67times theoretical
  FLOP reduction, and actual inference speedups of 1.76times, 1.85times,
  and 1.58times, respectively, while maintaining high visual fidelity, with
  PSNR values reaching 24.13, 27.09, and 22.59. Our work demonstrates that latent
  structural sparsity in vDiTs can be systematically exploited for long video
  synthesis.
paper_authors._id: 683fc6241de14546d5e02775
paper_authors.name: Pengtao Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T08:54:45.116Z
paper_authors._id: 64c9bac33cfe45b07179568d
paper_authors.avatarUrl: /avatars/4a8206cdb1770a8cdaae0d0a2b7b59f2.svg
paper_authors.fullname: Pengtao Chen
paper_authors.type: user
paper_authors.user: PengtaoChen
paper_authors._id: 683fc6241de14546d5e02776
paper_authors.name: Xianfang Zeng
paper_authors._id: 683fc6241de14546d5e02777
paper_authors.name: Maosen Zhao
paper_authors._id: 683fc6241de14546d5e02778
paper_authors.name: Peng Ye
paper_authors._id: 683fc6241de14546d5e02779
paper_authors.name: Mingzhu Shen
paper_authors._id: 683fc6241de14546d5e0277a
paper_authors.name: Wei Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T08:54:42.268Z
paper_authors._id: 64b914c8ace99c0723ad83a9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b914c8ace99c0723ad83a9/B4gxNByeVY_xaOcjwiN1j.jpeg
paper_authors.fullname: Wei Cheng
paper_authors.type: user
paper_authors.user: wchengad
paper_authors._id: 683fc6241de14546d5e0277b
paper_authors.name: Gang Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T08:54:39.068Z
paper_authors._id: 63417332c5565a4b8d43a0d8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63417332c5565a4b8d43a0d8/MmnYG7Wu2Z_lqCBvTfJmy.png
paper_authors.fullname: Gang Yu
paper_authors.type: user
paper_authors.user: skicy
paper_authors._id: 683fc6241de14546d5e0277c
paper_authors.name: Tao Chen
paper_title: Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate
    Video Diffusion Transformers
paper_summary: While Diffusion Transformers (DiTs) have achieved breakthroughs in video
  generation, this long sequence generation task remains constrained by the
  quadratic complexity of attention mechanisms, resulting in significant
  inference latency. Through detailed analysis of attention maps in Video
  Diffusion Transformer (vDiT), we identify three recurring sparsity patterns:
  diagonal, multi-diagonal, and vertical-stripe structures. And even 3-6\%
  attention heads can be skipped. Crucially, these patterns exhibit strong
  layer-depth and head-position correlations but show limited dependence on the
  input content. Leveraging these findings, we propose Sparse-vDiT, a sparsity
  acceleration framework for vDiT comprising: 1) Pattern-optimized sparse kernels
  that replace dense attention with computationally efficient implementations for
  each identified sparsity pattern. 2) An offline sparse diffusion search
  algorithm that selects the optimal sparse computation strategy per layer and
  head via hardware-aware cost modeling. After determining the optimal
  configuration, we fuse heads within the same layer that share the same
  attention strategy, enhancing inference efficiency. Integrated into
  state-of-the-art vDiT models (CogVideoX1.5, HunyuanVideo, and Wan2.1),
  Sparse-vDiT achieves 2.09times, 2.38times, and 1.67times theoretical
  FLOP reduction, and actual inference speedups of 1.76times, 1.85times,
  and 1.58times, respectively, while maintaining high visual fidelity, with
  PSNR values reaching 24.13, 27.09, and 22.59. Our work demonstrates that latent
  structural sparsity in vDiTs can be systematically exploited for long video
  synthesis.
paper_ai_summary: Sparse-vDiT accelerates Video Diffusion Transformer (vDiT) by leveraging sparsity patterns in attention maps, reducing theoretical FLOPs and improving inference speed without significant loss in visual quality.
paper_ai_keywords: Diffusion Transformers DiTs Video Diffusion Transformer vDiT sparsity patterns diagonal multi-diagonal vertical-stripe structures sparse kernels sparse diffusion search algorithm FLOP reduction inference speedups PSNR latent structural sparsity