title: FreeNoise: Tuning-Free Longer Video Diffusion Via Noise Rescheduling
summary: With the availability of large-scale video datasets and the advances of
  diffusion models, text-driven video generation has achieved substantial
  progress. However, existing video generation models are typically trained on a
  limited number of frames, resulting in the inability to generate high-fidelity
  long videos during inference. Furthermore, these models only support
  single-text conditions, whereas real-life scenarios often require multi-text
  conditions as the video content changes over time. To tackle these challenges,
  this study explores the potential of extending the text-driven capability to
  generate longer videos conditioned on multiple texts. 1) We first analyze the
  impact of initial noise in video diffusion models. Then building upon the
  observation of noise, we propose FreeNoise, a tuning-free and time-efficient
  paradigm to enhance the generative capabilities of pretrained video diffusion
  models while preserving content consistency. Specifically, instead of
  initializing noises for all frames, we reschedule a sequence of noises for
  long-range correlation and perform temporal attention over them by window-based
  function. 2) Additionally, we design a novel motion injection method to support
  the generation of videos conditioned on multiple text prompts. Extensive
  experiments validate the superiority of our paradigm in extending the
  generative capabilities of video diffusion models. It is noteworthy that
  compared with the previous best-performing method which brought about 255%
  extra time cost, our method incurs only negligible time cost of approximately
  17%. Generated video samples are available at our website:
  http://haonanqiu.com/projects/FreeNoise.html.
paper_authors._id: 65376a7a8ee17cfd440fb426
paper_authors.name: Haonan Qiu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:38:34.880Z
paper_authors._id: 63023b6ab002e9a4a2152890
paper_authors.avatarUrl: /avatars/cae8ba0a8d61fb4e576934431f43991b.svg
paper_authors.fullname: Haonan Qiu
paper_authors.type: user
paper_authors.user: MoonQiu
paper_authors._id: 65376a7a8ee17cfd440fb427
paper_authors.name: Menghan Xia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:38:55.514Z
paper_authors._id: 63401c89f81b9d101361f712
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665146415483-63401c89f81b9d101361f712.png
paper_authors.fullname: Richard
paper_authors.type: user
paper_authors.user: menghanxia
paper_authors._id: 65376a7a8ee17cfd440fb428
paper_authors.name: Yong Zhang
paper_authors._id: 65376a7a8ee17cfd440fb429
paper_authors.name: Yingqing He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:39:52.060Z
paper_authors._id: 630231bf7e137e3d6b3b0645
paper_authors.avatarUrl: /avatars/dad52f8955110f0a2caeb613d6aa3ea2.svg
paper_authors.fullname: He
paper_authors.type: user
paper_authors.user: Yingqing
paper_authors._id: 65376a7a8ee17cfd440fb42a
paper_authors.name: Xintao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:40:27.848Z
paper_authors._id: 60e272ca6c78a8c122b12127
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
paper_authors.fullname: Xintao Wang
paper_authors.type: user
paper_authors.user: Xintao
paper_authors._id: 65376a7a8ee17cfd440fb42b
paper_authors.name: Ying Shan
paper_authors._id: 65376a7a8ee17cfd440fb42c
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:40:51.757Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_title: FreeNoise: Tuning-Free Longer Video Diffusion Via Noise Rescheduling
paper_summary: With the availability of large-scale video datasets and the advances of
  diffusion models, text-driven video generation has achieved substantial
  progress. However, existing video generation models are typically trained on a
  limited number of frames, resulting in the inability to generate high-fidelity
  long videos during inference. Furthermore, these models only support
  single-text conditions, whereas real-life scenarios often require multi-text
  conditions as the video content changes over time. To tackle these challenges,
  this study explores the potential of extending the text-driven capability to
  generate longer videos conditioned on multiple texts. 1) We first analyze the
  impact of initial noise in video diffusion models. Then building upon the
  observation of noise, we propose FreeNoise, a tuning-free and time-efficient
  paradigm to enhance the generative capabilities of pretrained video diffusion
  models while preserving content consistency. Specifically, instead of
  initializing noises for all frames, we reschedule a sequence of noises for
  long-range correlation and perform temporal attention over them by window-based
  function. 2) Additionally, we design a novel motion injection method to support
  the generation of videos conditioned on multiple text prompts. Extensive
  experiments validate the superiority of our paradigm in extending the
  generative capabilities of video diffusion models. It is noteworthy that
  compared with the previous best-performing method which brought about 255%
  extra time cost, our method incurs only negligible time cost of approximately
  17%. Generated video samples are available at our website:
  http://haonanqiu.com/projects/FreeNoise.html.
paper_ai_keywords: diffusion models text-driven video generation high-fidelity long videos multi-text conditions FreeNoise time-efficient paradigm pretrained video diffusion models content consistency long-range correlation temporal attention window-based function motion injection method text prompts