title: Real-Time Video Generation with Pyramid Attention Broadcast
summary: We present Pyramid Attention Broadcast (PAB), a real-time, high quality and
  training-free approach for DiT-based video generation. Our method is founded on
  the observation that attention difference in the diffusion process exhibits a
  U-shaped pattern, indicating significant redundancy. We mitigate this by
  broadcasting attention outputs to subsequent steps in a pyramid style. It
  applies different broadcast strategies to each attention based on their
  variance for best efficiency. We further introduce broadcast sequence parallel
  for more efficient distributed inference. PAB demonstrates superior results
  across three models compared to baselines, achieving real-time generation for
  up to 720p videos. We anticipate that our simple yet effective method will
  serve as a robust baseline and facilitate future research and application for
  video generation.
paper_authors._id: 66c8073e0c80a52f46625e1d
paper_authors.name: Xuanlei Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-28T15:29:09.060Z
paper_authors._id: 64f5937db6d7050b19c68fec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64f5937db6d7050b19c68fec/4lWYw6-VxZsbl-1Rd9FSy.jpeg
paper_authors.fullname: Xuanlei Zhao
paper_authors.type: user
paper_authors.user: oahzxl
paper_authors._id: 66c8073e0c80a52f46625e1e
paper_authors.name: Xiaolong Jin
paper_authors._id: 66c8073e0c80a52f46625e1f
paper_authors.name: Kai Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-26T07:15:20.964Z
paper_authors._id: 655452b8432af1b1116394d1
paper_authors.avatarUrl: /avatars/85860fb3c2d09c9c23e7677d7129cca3.svg
paper_authors.fullname: Kai Wang
paper_authors.type: user
paper_authors.user: VictorKai1996NUS
paper_authors._id: 66c8073e0c80a52f46625e20
paper_authors.name: Yang You
paper_title: Real-Time Video Generation with Pyramid Attention Broadcast
paper_summary: We present Pyramid Attention Broadcast (PAB), a real-time, high quality and
  training-free approach for DiT-based video generation. Our method is founded on
  the observation that attention difference in the diffusion process exhibits a
  U-shaped pattern, indicating significant redundancy. We mitigate this by
  broadcasting attention outputs to subsequent steps in a pyramid style. It
  applies different broadcast strategies to each attention based on their
  variance for best efficiency. We further introduce broadcast sequence parallel
  for more efficient distributed inference. PAB demonstrates superior results
  across three models compared to baselines, achieving real-time generation for
  up to 720p videos. We anticipate that our simple yet effective method will
  serve as a robust baseline and facilitate future research and application for
  video generation.
paper_ai_keywords: Pyramid Attention Broadcast (PAB) attention difference diffusion process U-shaped pattern redundancy broadcasting attention outputs pyramid style broadcast strategies variance broadcast sequence parallel distributed inference DiT-based video generation