Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate
  Video Diffusion Transformers
While Diffusion Transformers (DiTs) have achieved breakthroughs in video
generation, this long sequence generation task remains constrained by the
quadratic complexity of attention mechanisms, resulting in significant
inference latency. Through detailed analysis of attention maps in Video
Diffusion Transformer (vDiT), we identify three recurring sparsity patterns:
diagonal, multi-diagonal, and vertical-stripe structures. And even 3-6\%
attention heads can be skipped. Crucially, these patterns exhibit strong
layer-depth and head-position correlations but show limited dependence on the
input content. Leveraging these findings, we propose Sparse-vDiT, a sparsity
acceleration framework for vDiT comprising: 1) Pattern-optimized sparse kernels
that replace dense attention with computationally efficient implementations for
each identified sparsity pattern. 2) An offline sparse diffusion search
algorithm that selects the optimal sparse computation strategy per layer and
head via hardware-aware cost modeling. After determining the optimal
configuration, we fuse heads within the same layer that share the same
attention strategy, enhancing inference efficiency. Integrated into
state-of-the-art vDiT models (CogVideoX1.5, HunyuanVideo, and Wan2.1),
Sparse-vDiT achieves 2.09times, 2.38times, and 1.67times theoretical
FLOP reduction, and actual inference speedups of 1.76times, 1.85times,
and 1.58times, respectively, while maintaining high visual fidelity, with
PSNR values reaching 24.13, 27.09, and 22.59. Our work demonstrates that latent
structural sparsity in vDiTs can be systematically exploited for long video
synthesis.
683fc6241de14546d5e02775
Pengtao Chen
claimed_verified
2025-06-04T08:54:45.116Z
64c9bac33cfe45b07179568d
/avatars/4a8206cdb1770a8cdaae0d0a2b7b59f2.svg
Pengtao Chen
user
PengtaoChen
683fc6241de14546d5e02776
Xianfang Zeng
683fc6241de14546d5e02777
Maosen Zhao
683fc6241de14546d5e02778
Peng Ye
683fc6241de14546d5e02779
Mingzhu Shen
683fc6241de14546d5e0277a
Wei Cheng
claimed_verified
2025-06-04T08:54:42.268Z
64b914c8ace99c0723ad83a9
https://cdn-avatars.huggingface.co/v1/production/uploads/64b914c8ace99c0723ad83a9/B4gxNByeVY_xaOcjwiN1j.jpeg
Wei Cheng
user
wchengad
683fc6241de14546d5e0277b
Gang Yu
claimed_verified
2025-06-04T08:54:39.068Z
63417332c5565a4b8d43a0d8
https://cdn-avatars.huggingface.co/v1/production/uploads/63417332c5565a4b8d43a0d8/MmnYG7Wu2Z_lqCBvTfJmy.png
Gang Yu
user
skicy
683fc6241de14546d5e0277c
Tao Chen
Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate
  Video Diffusion Transformers
While Diffusion Transformers (DiTs) have achieved breakthroughs in video
generation, this long sequence generation task remains constrained by the
quadratic complexity of attention mechanisms, resulting in significant
inference latency. Through detailed analysis of attention maps in Video
Diffusion Transformer (vDiT), we identify three recurring sparsity patterns:
diagonal, multi-diagonal, and vertical-stripe structures. And even 3-6\%
attention heads can be skipped. Crucially, these patterns exhibit strong
layer-depth and head-position correlations but show limited dependence on the
input content. Leveraging these findings, we propose Sparse-vDiT, a sparsity
acceleration framework for vDiT comprising: 1) Pattern-optimized sparse kernels
that replace dense attention with computationally efficient implementations for
each identified sparsity pattern. 2) An offline sparse diffusion search
algorithm that selects the optimal sparse computation strategy per layer and
head via hardware-aware cost modeling. After determining the optimal
configuration, we fuse heads within the same layer that share the same
attention strategy, enhancing inference efficiency. Integrated into
state-of-the-art vDiT models (CogVideoX1.5, HunyuanVideo, and Wan2.1),
Sparse-vDiT achieves 2.09times, 2.38times, and 1.67times theoretical
FLOP reduction, and actual inference speedups of 1.76times, 1.85times,
and 1.58times, respectively, while maintaining high visual fidelity, with
PSNR values reaching 24.13, 27.09, and 22.59. Our work demonstrates that latent
structural sparsity in vDiTs can be systematically exploited for long video
synthesis.
Sparse-vDiT accelerates Video Diffusion Transformer (vDiT) by leveraging sparsity patterns in attention maps, reducing theoretical FLOPs and improving inference speed without significant loss in visual quality.
Diffusion Transformers DiTs Video Diffusion Transformer vDiT sparsity patterns diagonal multi-diagonal vertical-stripe structures sparse kernels sparse diffusion search algorithm FLOP reduction inference speedups PSNR latent structural sparsity