VMoBA: Mixture-of-Block Attention for Video Diffusion Models
The quadratic complexity of full attention mechanisms poses a significant
bottleneck for Video Diffusion Models (VDMs) aiming to generate long-duration,
high-resolution videos. While various sparse attention methods have been
proposed, many are designed as training-free inference accelerators or do not
optimally capture the unique spatio-temporal characteristics inherent in video
data when trained natively. This paper introduces Video Mixture of Block
Attention (VMoBA), a novel sparse attention mechanism specifically adapted for
VDMs. Motivated by an in-depth analysis of attention patterns within
pre-trained video transformers, which revealed strong spatio-temporal locality,
varying query importance, and head-specific concentration levels, VMoBA
enhances the original MoBA framework with three key modifications: (1) a
layer-wise recurrent block partition scheme (1D-2D-3D) to dynamically adapt to
diverse spatio-temporal attention patterns and improve efficiency; (2) global
block selection to prioritize the most salient query-key block interactions
across an entire attention head; and (3) threshold-based block selection to
dynamically determine the number of attended blocks based on their cumulative
similarity. Extensive experiments demonstrate that VMoBA significantly
accelerates the training of VDMs on longer sequences, achieving 2.92x FLOPs and
1.48x latency speedup, while attaining comparable or even superior generation
quality to full attention. Furthermore, VMoBA exhibits competitive performance
in training-free inference, offering 2.40x FLOPs and 1.35x latency speedup for
high-res video generation.
686347d3588cea0da970c888
Jianzong Wu
686347d3588cea0da970c889
Liang Hou
claimed_verified
2025-07-01T10:13:49.312Z
64560a2aaaaf85a98fa9a4b9
https://cdn-avatars.huggingface.co/v1/production/uploads/64560a2aaaaf85a98fa9a4b9/2Kp0S0sMVpKqo81s-l_Yt.png
Liang Hou
user
lianghou
686347d3588cea0da970c88a
Haotian Yang
686347d3588cea0da970c88b
Xin Tao
686347d3588cea0da970c88c
Ye Tian
686347d3588cea0da970c88d
Pengfei Wan
686347d3588cea0da970c88e
Di Zhang
686347d3588cea0da970c88f
Yunhai Tong
VMoBA: Mixture-of-Block Attention for Video Diffusion Models
The quadratic complexity of full attention mechanisms poses a significant
bottleneck for Video Diffusion Models (VDMs) aiming to generate long-duration,
high-resolution videos. While various sparse attention methods have been
proposed, many are designed as training-free inference accelerators or do not
optimally capture the unique spatio-temporal characteristics inherent in video
data when trained natively. This paper introduces Video Mixture of Block
Attention (VMoBA), a novel sparse attention mechanism specifically adapted for
VDMs. Motivated by an in-depth analysis of attention patterns within
pre-trained video transformers, which revealed strong spatio-temporal locality,
varying query importance, and head-specific concentration levels, VMoBA
enhances the original MoBA framework with three key modifications: (1) a
layer-wise recurrent block partition scheme (1D-2D-3D) to dynamically adapt to
diverse spatio-temporal attention patterns and improve efficiency; (2) global
block selection to prioritize the most salient query-key block interactions
across an entire attention head; and (3) threshold-based block selection to
dynamically determine the number of attended blocks based on their cumulative
similarity. Extensive experiments demonstrate that VMoBA significantly
accelerates the training of VDMs on longer sequences, achieving 2.92x FLOPs and
1.48x latency speedup, while attaining comparable or even superior generation
quality to full attention. Furthermore, VMoBA exhibits competitive performance
in training-free inference, offering 2.40x FLOPs and 1.35x latency speedup for
high-res video generation.
VMoBA, a novel sparse attention mechanism, enhances Video Diffusion Models by improving efficiency and quality in generating long-duration, high-resolution videos.
Video Diffusion Models VDMs sparse attention Video Mixture of Block Attention VMoBA attention patterns video transformers spatio-temporal locality query importance head-specific concentration layer-wise recurrent block partition global block selection threshold-based block selection FLOPs latency speedup generation quality training-free inference