Real-Time Video Generation with Pyramid Attention Broadcast
We present Pyramid Attention Broadcast (PAB), a real-time, high quality and
training-free approach for DiT-based video generation. Our method is founded on
the observation that attention difference in the diffusion process exhibits a
U-shaped pattern, indicating significant redundancy. We mitigate this by
broadcasting attention outputs to subsequent steps in a pyramid style. It
applies different broadcast strategies to each attention based on their
variance for best efficiency. We further introduce broadcast sequence parallel
for more efficient distributed inference. PAB demonstrates superior results
across three models compared to baselines, achieving real-time generation for
up to 720p videos. We anticipate that our simple yet effective method will
serve as a robust baseline and facilitate future research and application for
video generation.
66c8073e0c80a52f46625e1d
Xuanlei Zhao
claimed_verified
2024-08-28T15:29:09.060Z
64f5937db6d7050b19c68fec
https://cdn-avatars.huggingface.co/v1/production/uploads/64f5937db6d7050b19c68fec/4lWYw6-VxZsbl-1Rd9FSy.jpeg
Xuanlei Zhao
user
oahzxl
66c8073e0c80a52f46625e1e
Xiaolong Jin
66c8073e0c80a52f46625e1f
Kai Wang
claimed_verified
2024-08-26T07:15:20.964Z
655452b8432af1b1116394d1
/avatars/85860fb3c2d09c9c23e7677d7129cca3.svg
Kai Wang
user
VictorKai1996NUS
66c8073e0c80a52f46625e20
Yang You
Real-Time Video Generation with Pyramid Attention Broadcast
We present Pyramid Attention Broadcast (PAB), a real-time, high quality and
training-free approach for DiT-based video generation. Our method is founded on
the observation that attention difference in the diffusion process exhibits a
U-shaped pattern, indicating significant redundancy. We mitigate this by
broadcasting attention outputs to subsequent steps in a pyramid style. It
applies different broadcast strategies to each attention based on their
variance for best efficiency. We further introduce broadcast sequence parallel
for more efficient distributed inference. PAB demonstrates superior results
across three models compared to baselines, achieving real-time generation for
up to 720p videos. We anticipate that our simple yet effective method will
serve as a robust baseline and facilitate future research and application for
video generation.
Pyramid Attention Broadcast (PAB) enhances DiT-based video generation by broadcasting attention outputs with a pyramid structure, achieving real-time performance and efficiency improvements.
Pyramid Attention Broadcast PAB DiT-based video generation diffusion process attention difference U-shaped pattern redundancy broadcast sequence parallel distributed inference real-time generation