title: Faster Video Diffusion with Trainable Sparse Attention
summary: Scaling video diffusion transformers (DiTs) is limited by their quadratic 3D
  attention, even though most of the attention mass concentrates on a small
  subset of positions. We turn this observation into VSA, a trainable,
  hardware-efficient sparse attention that replaces full attention at both
  training and inference. In VSA, a lightweight coarse stage pools tokens into
  tiles and identifies high-weight critical tokens; a fine stage computes
  token-level attention only inside those tiles subjecting to block computing
  layout to ensure hard efficiency. This leads to a single differentiable kernel
  that trains end-to-end, requires no post-hoc profiling, and sustains 85\% of
  FlashAttention3 MFU. We perform a large sweep of ablation studies and
  scaling-law experiments by pretraining DiTs from 60M to 1.4B parameters. VSA
  reaches a Pareto point that cuts training FLOPS by 2.53times with no drop in
  diffusion loss. Retrofitting the open-source Wan-2.1 model speeds up attention
  time by 6times and lowers end-to-end generation time from 31s to 18s with
  comparable quality. These results establish trainable sparse attention as a
  practical alternative to full attention and a key enabler for further scaling
  of video diffusion models.
paper_authors._id: 682c27e2fffb36958f8cd84e
paper_authors.name: Peiyuan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:36:24.007Z
paper_authors._id: 63565cc56d7fcf1bedb7d347
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63565cc56d7fcf1bedb7d347/XGcHP4VkO_oieA1gZ4IAX.jpeg
paper_authors.fullname: Zhang Peiyuan
paper_authors.type: user
paper_authors.user: PY007
paper_authors._id: 682c27e2fffb36958f8cd84f
paper_authors.name: Haofeng Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:36:29.763Z
paper_authors._id: 67ea1f6693f71dd8167a2d22
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/H_upra_XVG1AoBKUe9ArV.png
paper_authors.fullname: haofeng huang
paper_authors.type: user
paper_authors.user: haofeng666
paper_authors._id: 682c27e2fffb36958f8cd850
paper_authors.name: Yongqi Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:36:36.258Z
paper_authors._id: 65416817271d3bc4d70f6745
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65416817271d3bc4d70f6745/1YkW0MpuufejvxqksVMIx.jpeg
paper_authors.fullname: Yongqi Chen
paper_authors.type: user
paper_authors.user: BrianChen1129
paper_authors._id: 682c27e2fffb36958f8cd851
paper_authors.name: Will Lin
paper_authors._id: 682c27e2fffb36958f8cd852
paper_authors.name: Zhengzhong Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:36:46.746Z
paper_authors._id: 62fbdc67c776fd8821ae3f2d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62fbdc67c776fd8821ae3f2d/cI7iAZOL40RUYluo5ZVTU.png
paper_authors.fullname: Zhengzhong Liu
paper_authors.type: user
paper_authors.user: hunterhector
paper_authors._id: 682c27e2fffb36958f8cd853
paper_authors.name: Ion Stoica
paper_authors._id: 682c27e2fffb36958f8cd854
paper_authors.name: Eric P. Xing
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:37:02.570Z
paper_authors._id: 64ff67722ad36636be6c4542
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sLIrNelAWPVOy4e3oo5LB.jpeg
paper_authors.fullname: Eric Xing
paper_authors.type: user
paper_authors.user: EricX003
paper_authors._id: 682c27e2fffb36958f8cd855
paper_authors.name: Hao Zhang
paper_title: Faster Video Diffusion with Trainable Sparse Attention
paper_summary: Scaling video diffusion transformers (DiTs) is limited by their quadratic 3D
  attention, even though most of the attention mass concentrates on a small
  subset of positions. We turn this observation into VSA, a trainable,
  hardware-efficient sparse attention that replaces full attention at both
  training and inference. In VSA, a lightweight coarse stage pools tokens into
  tiles and identifies high-weight critical tokens; a fine stage computes
  token-level attention only inside those tiles subjecting to block computing
  layout to ensure hard efficiency. This leads to a single differentiable kernel
  that trains end-to-end, requires no post-hoc profiling, and sustains 85\% of
  FlashAttention3 MFU. We perform a large sweep of ablation studies and
  scaling-law experiments by pretraining DiTs from 60M to 1.4B parameters. VSA
  reaches a Pareto point that cuts training FLOPS by 2.53times with no drop in
  diffusion loss. Retrofitting the open-source Wan-2.1 model speeds up attention
  time by 6times and lowers end-to-end generation time from 31s to 18s with
  comparable quality. These results establish trainable sparse attention as a
  practical alternative to full attention and a key enabler for further scaling
  of video diffusion models.
paper_ai_summary: Trainable sparse attention (VSA) reduces computational cost in video diffusion transformers with minimal impact on performance, enabling efficient scaling of the models.
paper_ai_keywords: video diffusion transformers DiTs quadratic 3D attention VSA token-level attention block computing FlashAttention3 MFU FLOPS diffusion loss Wan-2.1 model