BroadWay: Boost Your Text-to-Video Generation Model in a Training-free
  Way
The text-to-video (T2V) generation models, offering convenient visual
creation, have recently garnered increasing attention. Despite their
substantial potential, the generated videos may present artifacts, including
structural implausibility, temporal inconsistency, and a lack of motion, often
resulting in near-static video. In this work, we have identified a correlation
between the disparity of temporal attention maps across different blocks and
the occurrence of temporal inconsistencies. Additionally, we have observed that
the energy contained within the temporal attention maps is directly related to
the magnitude of motion amplitude in the generated videos. Based on these
observations, we present BroadWay, a training-free method to improve the
quality of text-to-video generation without introducing additional parameters,
augmenting memory or sampling time. Specifically, BroadWay is composed of two
principal components: 1) Temporal Self-Guidance improves the structural
plausibility and temporal consistency of generated videos by reducing the
disparity between the temporal attention maps across various decoder blocks. 2)
Fourier-based Motion Enhancement enhances the magnitude and richness of motion
by amplifying the energy of the map. Extensive experiments demonstrate that
BroadWay significantly improves the quality of text-to-video generation with
negligible additional cost.
670749315ea108eecd14ec6f
Jiazi Bu
670749315ea108eecd14ec70
Pengyang Ling
670749315ea108eecd14ec71
Pan Zhang
670749315ea108eecd14ec72
Tong Wu
claimed_verified
2024-12-11T09:21:50.443Z
6310929fda0aaad2f70397a1
https://cdn-avatars.huggingface.co/v1/production/uploads/6310929fda0aaad2f70397a1/YoHeYZezmCOXwL-qsiZOc.png
Tong Wu
user
wutong16
670749315ea108eecd14ec73
Xiaoyi Dong
670749315ea108eecd14ec74
Yuhang Zang
admin_assigned
2024-10-10T09:56:05.632Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/mCgynEtoXdILvzyoPexii.png
Yuhang Zang
user
yuhangzang
670749315ea108eecd14ec75
Yuhang Cao
670749315ea108eecd14ec76
Dahua Lin
admin_assigned
2024-10-10T09:56:35.238Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
670749315ea108eecd14ec77
Jiaqi Wang
claimed_verified
2024-12-11T09:21:48.196Z
64b4eec4faa3181a5eab9c46
/avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
Jiaqi Wang
user
myownskyW7
BroadWay: Boost Your Text-to-Video Generation Model in a Training-free
  Way
The text-to-video (T2V) generation models, offering convenient visual
creation, have recently garnered increasing attention. Despite their
substantial potential, the generated videos may present artifacts, including
structural implausibility, temporal inconsistency, and a lack of motion, often
resulting in near-static video. In this work, we have identified a correlation
between the disparity of temporal attention maps across different blocks and
the occurrence of temporal inconsistencies. Additionally, we have observed that
the energy contained within the temporal attention maps is directly related to
the magnitude of motion amplitude in the generated videos. Based on these
observations, we present BroadWay, a training-free method to improve the
quality of text-to-video generation without introducing additional parameters,
augmenting memory or sampling time. Specifically, BroadWay is composed of two
principal components: 1) Temporal Self-Guidance improves the structural
plausibility and temporal consistency of generated videos by reducing the
disparity between the temporal attention maps across various decoder blocks. 2)
Fourier-based Motion Enhancement enhances the magnitude and richness of motion
by amplifying the energy of the map. Extensive experiments demonstrate that
BroadWay significantly improves the quality of text-to-video generation with
negligible additional cost.
BroadWay, a training-free method, improves text-to-video generation by addressing temporal inconsistencies and enhancing motion through temporal self-guidance and Fourier-based motion enhancement.
text-to-video T2V temporal attention maps structural plausibility temporal consistency motion amplitude Temporal Self-Guidance Fourier-based Motion Enhancement