title: Fast Video Generation with Sliding Tile Attention
summary: Diffusion Transformers (DiTs) with 3D full attention power state-of-the-art
  video generation, but suffer from prohibitive compute cost -- when generating
  just a 5-second 720P video, attention alone takes 800 out of 945 seconds of
  total inference time. This paper introduces sliding tile attention (STA) to
  address this challenge. STA leverages the observation that attention scores in
  pretrained video diffusion models predominantly concentrate within localized 3D
  windows. By sliding and attending over the local spatial-temporal region, STA
  eliminates redundancy from full attention. Unlike traditional token-wise
  sliding window attention (SWA), STA operates tile-by-tile with a novel
  hardware-aware sliding window design, preserving expressiveness while being
  hardware-efficient. With careful kernel-level optimizations, STA offers the
  first efficient 2D/3D sliding-window-like attention implementation, achieving
  58.79% MFU. Precisely, STA accelerates attention by 2.8-17x over
  FlashAttention-2 (FA2) and 1.6-10x over FlashAttention-3 (FA3). On the leading
  video DiT, HunyuanVideo, STA reduces end-to-end latency from 945s (FA3) to 685s
  without quality degradation, requiring no training. Enabling finetuning further
  lowers latency to 268s with only a 0.09% drop on VBench.
paper_authors._id: 67a98cd1b8b21202c9004628
paper_authors.name: Peiyuan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-10T16:07:27.309Z
paper_authors._id: 63565cc56d7fcf1bedb7d347
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63565cc56d7fcf1bedb7d347/XGcHP4VkO_oieA1gZ4IAX.jpeg
paper_authors.fullname: Zhang Peiyuan
paper_authors.type: user
paper_authors.user: PY007
paper_authors._id: 67a98cd1b8b21202c9004629
paper_authors.name: Yongqi Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-10T09:49:48.410Z
paper_authors._id: 65416817271d3bc4d70f6745
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65416817271d3bc4d70f6745/1YkW0MpuufejvxqksVMIx.jpeg
paper_authors.fullname: Yongqi Chen
paper_authors.type: user
paper_authors.user: BrianChen1129
paper_authors._id: 67a98cd1b8b21202c900462a
paper_authors.name: Runlong Su
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T10:00:40.942Z
paper_authors._id: 65d7ed4823e83e1591beacc7
paper_authors.avatarUrl: /avatars/2a6714a2a7bbd591f6b726a7330bafbc.svg
paper_authors.fullname: Su
paper_authors.type: user
paper_authors.user: r3su9
paper_authors._id: 67a98cd1b8b21202c900462b
paper_authors.name: Hangliang Ding
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T07:56:04.110Z
paper_authors._id: 643a451ee2b979ae6141329d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643a451ee2b979ae6141329d/HN3M5vyroanQoUEiXJFyB.jpeg
paper_authors.fullname: Hangliang Ding
paper_authors.type: user
paper_authors.user: foreverpiano
paper_authors._id: 67a98cd1b8b21202c900462c
paper_authors.name: Ion Stoica
paper_authors._id: 67a98cd1b8b21202c900462d
paper_authors.name: Zhenghong Liu
paper_authors._id: 67a98cd1b8b21202c900462e
paper_authors.name: Hao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T10:00:38.019Z
paper_authors._id: 62d363143eebd640a4fa41fa
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62d363143eebd640a4fa41fa/pvPwXlJ5OOb-UIfmffv4E.jpeg
paper_authors.fullname: Hao Zhang
paper_authors.type: user
paper_authors.user: zhisbug
paper_title: Fast Video Generation with Sliding Tile Attention
paper_summary: Diffusion Transformers (DiTs) with 3D full attention power state-of-the-art
  video generation, but suffer from prohibitive compute cost -- when generating
  just a 5-second 720P video, attention alone takes 800 out of 945 seconds of
  total inference time. This paper introduces sliding tile attention (STA) to
  address this challenge. STA leverages the observation that attention scores in
  pretrained video diffusion models predominantly concentrate within localized 3D
  windows. By sliding and attending over the local spatial-temporal region, STA
  eliminates redundancy from full attention. Unlike traditional token-wise
  sliding window attention (SWA), STA operates tile-by-tile with a novel
  hardware-aware sliding window design, preserving expressiveness while being
  hardware-efficient. With careful kernel-level optimizations, STA offers the
  first efficient 2D/3D sliding-window-like attention implementation, achieving
  58.79% MFU. Precisely, STA accelerates attention by 2.8-17x over
  FlashAttention-2 (FA2) and 1.6-10x over FlashAttention-3 (FA3). On the leading
  video DiT, HunyuanVideo, STA reduces end-to-end latency from 945s (FA3) to 685s
  without quality degradation, requiring no training. Enabling finetuning further
  lowers latency to 268s with only a 0.09% drop on VBench.
paper_ai_keywords: Diffusion Transformers (DiTs) 3D full attention video generation sliding tile attention (STA) attention scores pretrained video diffusion models 3D windows sliding window attention (SWA) hardware-aware sliding window design FlashAttention-2 (FA2) FlashAttention-3 (FA3) VBench HunyuanVideo