Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via
  Semantic-Aware Permutation
Diffusion Transformers (DiTs) are essential for video generation but suffer
from significant latency due to the quadratic complexity of attention. By
computing only critical tokens, sparse attention reduces computational costs
and offers a promising acceleration approach. However, we identify that
existing methods fail to approach optimal generation quality under the same
computation budget for two reasons: (1) Inaccurate critical token
identification: current methods cluster tokens based on position rather than
semantics, leading to imprecise aggregated representations. (2) Excessive
computation waste: critical tokens are scattered among non-critical ones,
leading to wasted computation on GPUs, which are optimized for processing
contiguous tokens. In this paper, we propose SVG2, a training-free framework
that maximizes identification accuracy and minimizes computation waste,
achieving a Pareto frontier trade-off between generation quality and
efficiency. The core of SVG2 is semantic-aware permutation, which clusters and
reorders tokens based on semantic similarity using k-means. This approach
ensures both a precise cluster representation, improving identification
accuracy, and a densified layout of critical tokens, enabling efficient
computation without padding. Additionally, SVG2 integrates top-p dynamic budget
control and customized kernel implementations, achieving up to 2.30x and 1.89x
speedup while maintaining a PSNR of up to 30 and 26 on HunyuanVideo and Wan
2.1, respectively.
683536db6d3dc82656b13765
Shuo Yang
claimed_verified
2025-05-27T07:50:23.533Z
642b970ceb31218a5f204a29
/avatars/582287f477bbb1a0842787145e375fd3.svg
andy-yang
user
andy-yang
683536db6d3dc82656b13766
Haocheng Xi
claimed_verified
2025-05-27T07:49:30.035Z
66ce751a8ec9fda2cf5a9e85
/avatars/c17093ca81dad007b3e50bae503955a7.svg
Haocheng Xi
user
xihc-ucb
683536db6d3dc82656b13767
Yilong Zhao
claimed_verified
2025-05-27T07:49:32.515Z
6549b0a808775ce78e535c6a
/avatars/942066356843d0c424375937f157c975.svg
Yilong Zhao
user
ylzhao
683536db6d3dc82656b13768
Muyang Li
claimed_verified
2025-05-29T07:44:05.243Z
63129589bbaa385279d1826e
https://cdn-avatars.huggingface.co/v1/production/uploads/63129589bbaa385279d1826e/0AUs3q4ngRZ-wXuY1jP9G.jpeg
Muyang Li
user
Lmxyy
683536db6d3dc82656b13769
Jintao Zhang
claimed_verified
2025-05-28T09:00:28.809Z
66c0a08bac74db25de8427ec
https://cdn-avatars.huggingface.co/v1/production/uploads/66c0a08bac74db25de8427ec/9D-piDBZqSt6KNkHImmkv.jpeg
Jintao Zhang
user
jt-zhang
683536db6d3dc82656b1376a
Han Cai
claimed_verified
2025-05-28T09:00:30.674Z
650e2b14c945dfc9386a7e28
/avatars/0a9be20aa53f4c52a2d1a8b02d4093ea.svg
Han Cai
user
han-cai
683536db6d3dc82656b1376b
Yujun Lin
683536db6d3dc82656b1376c
Xiuyu Li
683536db6d3dc82656b1376d
Chenfeng Xu
claimed_verified
2025-06-04T09:04:22.005Z
654ca71d5255ee86711b52c5
/avatars/52bf00fd74c8db5643c4daa185c678e6.svg
Chenfeng Xu
user
chenfengx
683536db6d3dc82656b1376e
Kelly Peng
683536db6d3dc82656b1376f
Jianfei Chen
683536db6d3dc82656b13770
Song Han
683536db6d3dc82656b13771
Kurt Keutzer
683536db6d3dc82656b13772
Ion Stoica
Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via
  Semantic-Aware Permutation
Diffusion Transformers (DiTs) are essential for video generation but suffer
from significant latency due to the quadratic complexity of attention. By
computing only critical tokens, sparse attention reduces computational costs
and offers a promising acceleration approach. However, we identify that
existing methods fail to approach optimal generation quality under the same
computation budget for two reasons: (1) Inaccurate critical token
identification: current methods cluster tokens based on position rather than
semantics, leading to imprecise aggregated representations. (2) Excessive
computation waste: critical tokens are scattered among non-critical ones,
leading to wasted computation on GPUs, which are optimized for processing
contiguous tokens. In this paper, we propose SVG2, a training-free framework
that maximizes identification accuracy and minimizes computation waste,
achieving a Pareto frontier trade-off between generation quality and
efficiency. The core of SVG2 is semantic-aware permutation, which clusters and
reorders tokens based on semantic similarity using k-means. This approach
ensures both a precise cluster representation, improving identification
accuracy, and a densified layout of critical tokens, enabling efficient
computation without padding. Additionally, SVG2 integrates top-p dynamic budget
control and customized kernel implementations, achieving up to 2.30x and 1.89x
speedup while maintaining a PSNR of up to 30 and 26 on HunyuanVideo and Wan
2.1, respectively.
SVG2 is a training-free framework that enhances video generation efficiency and quality by accurately identifying and processing critical tokens using semantic-aware permutation and dynamic budget control.
Diffusion Transformers sparse attention critical tokens semantic similarity semantic-aware permutation k-means top-p dynamic budget control