FancyVideo: Towards Dynamic and Consistent Video Generation via
  Cross-frame Textual Guidance
Synthesizing motion-rich and temporally consistent videos remains a challenge
in artificial intelligence, especially when dealing with extended durations.
Existing text-to-video (T2V) models commonly employ spatial cross-attention for
text control, equivalently guiding different frame generations without
frame-specific textual guidance. Thus, the model's capacity to comprehend the
temporal logic conveyed in prompts and generate videos with coherent motion is
restricted. To tackle this limitation, we introduce FancyVideo, an innovative
video generator that improves the existing text-control mechanism with the
well-designed Cross-frame Textual Guidance Module (CTGM). Specifically, CTGM
incorporates the Temporal Information Injector (TII), Temporal Affinity Refiner
(TAR), and Temporal Feature Booster (TFB) at the beginning, middle, and end of
cross-attention, respectively, to achieve frame-specific textual guidance.
Firstly, TII injects frame-specific information from latent features into text
conditions, thereby obtaining cross-frame textual conditions. Then, TAR refines
the correlation matrix between cross-frame textual conditions and latent
features along the time dimension. Lastly, TFB boosts the temporal consistency
of latent features. Extensive experiments comprising both quantitative and
qualitative evaluations demonstrate the effectiveness of FancyVideo. Our
approach achieves state-of-the-art T2V generation results on the EvalCrafter
benchmark and facilitates the synthesis of dynamic and consistent videos. The
video show results can be available at https://fancyvideo.github.io/, and we
will make our code and model weights publicly available.
66beac255f4422f88654b79a
Jiasong Feng
claimed_verified
2024-09-10T08:38:01.848Z
65542ae4520d9bb9d9d21659
/avatars/c89f2e76f2f1f4e162d15f4da8280f19.svg
js
user
feng8
66beac255f4422f88654b79b
Ao Ma
claimed_verified
2024-09-11T07:27:13.427Z
65292034825f4bba46c78581
https://cdn-avatars.huggingface.co/v1/production/uploads/65292034825f4bba46c78581/8pkETWFhha7AUM6E37Rvl.jpeg
AoMa
user
AoMa
66beac255f4422f88654b79c
Jing Wang
66beac255f4422f88654b79d
Bo Cheng
66beac255f4422f88654b79e
Xiaodan Liang
66beac255f4422f88654b79f
Dawei Leng
extracted_confirmed
2024-10-18T02:23:18.427Z
649935abbe8fd92c27ab1ed8
https://cdn-avatars.huggingface.co/v1/production/uploads/649935abbe8fd92c27ab1ed8/ueWnaZtJa-oWpzupP6FV8.png
David Leon
user
DavidLeon
66beac255f4422f88654b7a0
Yuhui Yin
FancyVideo: Towards Dynamic and Consistent Video Generation via
  Cross-frame Textual Guidance
Synthesizing motion-rich and temporally consistent videos remains a challenge
in artificial intelligence, especially when dealing with extended durations.
Existing text-to-video (T2V) models commonly employ spatial cross-attention for
text control, equivalently guiding different frame generations without
frame-specific textual guidance. Thus, the model's capacity to comprehend the
temporal logic conveyed in prompts and generate videos with coherent motion is
restricted. To tackle this limitation, we introduce FancyVideo, an innovative
video generator that improves the existing text-control mechanism with the
well-designed Cross-frame Textual Guidance Module (CTGM). Specifically, CTGM
incorporates the Temporal Information Injector (TII), Temporal Affinity Refiner
(TAR), and Temporal Feature Booster (TFB) at the beginning, middle, and end of
cross-attention, respectively, to achieve frame-specific textual guidance.
Firstly, TII injects frame-specific information from latent features into text
conditions, thereby obtaining cross-frame textual conditions. Then, TAR refines
the correlation matrix between cross-frame textual conditions and latent
features along the time dimension. Lastly, TFB boosts the temporal consistency
of latent features. Extensive experiments comprising both quantitative and
qualitative evaluations demonstrate the effectiveness of FancyVideo. Our
approach achieves state-of-the-art T2V generation results on the EvalCrafter
benchmark and facilitates the synthesis of dynamic and consistent videos. The
video show results can be available at https://fancyvideo.github.io/, and we
will make our code and model weights publicly available.
FancyVideo introduces the CTGM to improve temporal consistency in text-to-video synthesis by incorporating TII, TAR, and TFB for frame-specific text control.
Cross-frame Textual Guidance Module CTGM Temporal Information Injector TII Temporal Affinity Refiner TAR Temporal Feature Booster TFB text-to-video T2V latency features cross-attention EvalCrafter benchmark