title: VSTAR: Generative Temporal Nursing for Longer Dynamic Video Synthesis
summary: Despite tremendous progress in the field of text-to-video (T2V) synthesis,
  open-sourced T2V diffusion models struggle to generate longer videos with
  dynamically varying and evolving content. They tend to synthesize quasi-static
  videos, ignoring the necessary visual change-over-time implied in the text
  prompt. At the same time, scaling these models to enable longer, more dynamic
  video synthesis often remains computationally intractable. To address this
  challenge, we introduce the concept of Generative Temporal Nursing (GTN), where
  we aim to alter the generative process on the fly during inference to improve
  control over the temporal dynamics and enable generation of longer videos. We
  propose a method for GTN, dubbed VSTAR, which consists of two key ingredients:
  1) Video Synopsis Prompting (VSP) - automatic generation of a video synopsis
  based on the original single prompt leveraging LLMs, which gives accurate
  textual guidance to different visual states of longer videos, and 2) Temporal
  Attention Regularization (TAR) - a regularization technique to refine the
  temporal attention units of the pre-trained T2V diffusion models, which enables
  control over the video dynamics. We experimentally showcase the superiority of
  the proposed approach in generating longer, visually appealing videos over
  existing open-sourced T2V models. We additionally analyze the temporal
  attention maps realized with and without VSTAR, demonstrating the importance of
  applying our method to mitigate neglect of the desired visual change over time.
paper_authors._id: 65fbb4febeb503364b8210fd
paper_authors.name: Yumeng Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-21T10:21:19.202Z
paper_authors._id: 62a8e65cbad2c679dc0b2866
paper_authors.avatarUrl: /avatars/08f06947880f74145005f0da1a82dfed.svg
paper_authors.fullname: Yumeng Li
paper_authors.type: user
paper_authors.user: Yumeng
paper_authors._id: 65fbb4febeb503364b8210fe
paper_authors.name: William Beluch
paper_authors._id: 65fbb4febeb503364b8210ff
paper_authors.name: Margret Keuper
paper_authors._id: 65fbb4febeb503364b821100
paper_authors.name: Dan Zhang
paper_authors._id: 65fbb4febeb503364b821101
paper_authors.name: Anna Khoreva
paper_title: VSTAR: Generative Temporal Nursing for Longer Dynamic Video Synthesis
paper_summary: Despite tremendous progress in the field of text-to-video (T2V) synthesis,
  open-sourced T2V diffusion models struggle to generate longer videos with
  dynamically varying and evolving content. They tend to synthesize quasi-static
  videos, ignoring the necessary visual change-over-time implied in the text
  prompt. At the same time, scaling these models to enable longer, more dynamic
  video synthesis often remains computationally intractable. To address this
  challenge, we introduce the concept of Generative Temporal Nursing (GTN), where
  we aim to alter the generative process on the fly during inference to improve
  control over the temporal dynamics and enable generation of longer videos. We
  propose a method for GTN, dubbed VSTAR, which consists of two key ingredients:
  1) Video Synopsis Prompting (VSP) - automatic generation of a video synopsis
  based on the original single prompt leveraging LLMs, which gives accurate
  textual guidance to different visual states of longer videos, and 2) Temporal
  Attention Regularization (TAR) - a regularization technique to refine the
  temporal attention units of the pre-trained T2V diffusion models, which enables
  control over the video dynamics. We experimentally showcase the superiority of
  the proposed approach in generating longer, visually appealing videos over
  existing open-sourced T2V models. We additionally analyze the temporal
  attention maps realized with and without VSTAR, demonstrating the importance of
  applying our method to mitigate neglect of the desired visual change over time.
paper_ai_keywords: diffusion models text-to-video (T2V) synthesis visually dynamic content generative process temporal dynamics Generative Temporal Nursing (GTN) VSTAR Video Synopsis Prompting (VSP) Large Language Models (LLMs) Temporal Performance Regularization (TAR) temporal attention maps pre-trained T2V diffusion models