title: FancyVideo: Towards Dynamic and Consistent Video Generation via
    Cross-frame Textual Guidance
summary: Synthesizing motion-rich and temporally consistent videos remains a challenge
  in artificial intelligence, especially when dealing with extended durations.
  Existing text-to-video (T2V) models commonly employ spatial cross-attention for
  text control, equivalently guiding different frame generations without
  frame-specific textual guidance. Thus, the model's capacity to comprehend the
  temporal logic conveyed in prompts and generate videos with coherent motion is
  restricted. To tackle this limitation, we introduce FancyVideo, an innovative
  video generator that improves the existing text-control mechanism with the
  well-designed Cross-frame Textual Guidance Module (CTGM). Specifically, CTGM
  incorporates the Temporal Information Injector (TII), Temporal Affinity Refiner
  (TAR), and Temporal Feature Booster (TFB) at the beginning, middle, and end of
  cross-attention, respectively, to achieve frame-specific textual guidance.
  Firstly, TII injects frame-specific information from latent features into text
  conditions, thereby obtaining cross-frame textual conditions. Then, TAR refines
  the correlation matrix between cross-frame textual conditions and latent
  features along the time dimension. Lastly, TFB boosts the temporal consistency
  of latent features. Extensive experiments comprising both quantitative and
  qualitative evaluations demonstrate the effectiveness of FancyVideo. Our
  approach achieves state-of-the-art T2V generation results on the EvalCrafter
  benchmark and facilitates the synthesis of dynamic and consistent videos. The
  video show results can be available at https://fancyvideo.github.io/, and we
  will make our code and model weights publicly available.
paper_authors._id: 66beac255f4422f88654b79a
paper_authors.name: Jiasong Feng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-10T08:38:01.848Z
paper_authors._id: 65542ae4520d9bb9d9d21659
paper_authors.avatarUrl: /avatars/c89f2e76f2f1f4e162d15f4da8280f19.svg
paper_authors.fullname: js
paper_authors.type: user
paper_authors.user: feng8
paper_authors._id: 66beac255f4422f88654b79b
paper_authors.name: Ao Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-11T07:27:13.427Z
paper_authors._id: 65292034825f4bba46c78581
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65292034825f4bba46c78581/8pkETWFhha7AUM6E37Rvl.jpeg
paper_authors.fullname: AoMa
paper_authors.type: user
paper_authors.user: AoMa
paper_authors._id: 66beac255f4422f88654b79c
paper_authors.name: Jing Wang
paper_authors._id: 66beac255f4422f88654b79d
paper_authors.name: Bo Cheng
paper_authors._id: 66beac255f4422f88654b79e
paper_authors.name: Xiaodan Liang
paper_authors._id: 66beac255f4422f88654b79f
paper_authors.name: Dawei Leng
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-18T02:23:18.427Z
paper_authors._id: 649935abbe8fd92c27ab1ed8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649935abbe8fd92c27ab1ed8/ueWnaZtJa-oWpzupP6FV8.png
paper_authors.fullname: David Leon
paper_authors.type: user
paper_authors.user: DavidLeon
paper_authors._id: 66beac255f4422f88654b7a0
paper_authors.name: Yuhui Yin
paper_title: FancyVideo: Towards Dynamic and Consistent Video Generation via
    Cross-frame Textual Guidance
paper_summary: Synthesizing motion-rich and temporally consistent videos remains a challenge
  in artificial intelligence, especially when dealing with extended durations.
  Existing text-to-video (T2V) models commonly employ spatial cross-attention for
  text control, equivalently guiding different frame generations without
  frame-specific textual guidance. Thus, the model's capacity to comprehend the
  temporal logic conveyed in prompts and generate videos with coherent motion is
  restricted. To tackle this limitation, we introduce FancyVideo, an innovative
  video generator that improves the existing text-control mechanism with the
  well-designed Cross-frame Textual Guidance Module (CTGM). Specifically, CTGM
  incorporates the Temporal Information Injector (TII), Temporal Affinity Refiner
  (TAR), and Temporal Feature Booster (TFB) at the beginning, middle, and end of
  cross-attention, respectively, to achieve frame-specific textual guidance.
  Firstly, TII injects frame-specific information from latent features into text
  conditions, thereby obtaining cross-frame textual conditions. Then, TAR refines
  the correlation matrix between cross-frame textual conditions and latent
  features along the time dimension. Lastly, TFB boosts the temporal consistency
  of latent features. Extensive experiments comprising both quantitative and
  qualitative evaluations demonstrate the effectiveness of FancyVideo. Our
  approach achieves state-of-the-art T2V generation results on the EvalCrafter
  benchmark and facilitates the synthesis of dynamic and consistent videos. The
  video show results can be available at https://fancyvideo.github.io/, and we
  will make our code and model weights publicly available.
paper_ai_summary: FancyVideo introduces the CTGM to improve temporal consistency in text-to-video synthesis by incorporating TII, TAR, and TFB for frame-specific text control.
paper_ai_keywords: Cross-frame Textual Guidance Module CTGM Temporal Information Injector TII Temporal Affinity Refiner TAR Temporal Feature Booster TFB text-to-video T2V latency features cross-attention EvalCrafter benchmark