title: Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models
summary: Despite tremendous progress in generating high-quality images using diffusion
  models, synthesizing a sequence of animated frames that are both photorealistic
  and temporally coherent is still in its infancy. While off-the-shelf
  billion-scale datasets for image generation are available, collecting similar
  video data of the same scale is still challenging. Also, training a video
  diffusion model is computationally much more expensive than its image
  counterpart. In this work, we explore finetuning a pretrained image diffusion
  model with video data as a practical solution for the video synthesis task. We
  find that naively extending the image noise prior to video noise prior in video
  diffusion leads to sub-optimal performance. Our carefully designed video noise
  prior leads to substantially better performance. Extensive experimental
  validation shows that our model, Preserve Your Own Correlation (PYoCo), attains
  SOTA zero-shot text-to-video results on the UCF-101 and MSR-VTT benchmarks. It
  also achieves SOTA video generation quality on the small-scale UCF-101
  benchmark with a 10times smaller model using significantly less computation
  than the prior art.
paper_authors._id: 6466d620060756d2854abb76
paper_authors.name: Songwei Ge
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-23T08:11:29.465Z
paper_authors._id: 631cdc696e623740a78fd864
paper_authors.avatarUrl: /avatars/f610873f6080453b655fb230d6f3b765.svg
paper_authors.fullname: Songwei Ge
paper_authors.type: user
paper_authors.user: songweig
paper_authors._id: 6466d620060756d2854abb77
paper_authors.name: Seungjun Nah
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-21T20:34:11.354Z
paper_authors._id: 62f6d3278465979f627e00ff
paper_authors.avatarUrl: /avatars/4b96c355e6d8e9db0a23eaf85da7b215.svg
paper_authors.fullname: Seungjun Nah
paper_authors.type: user
paper_authors.user: snah
paper_authors._id: 6466d620060756d2854abb78
paper_authors.name: Guilin Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-22T10:18:43.129Z
paper_authors._id: 63bc4115141c7d395c4aa02c
paper_authors.avatarUrl: /avatars/a9eca59e8c52cceab6524a4aaab11e38.svg
paper_authors.fullname: Liu
paper_authors.type: user
paper_authors.user: liuguilin
paper_authors._id: 6466d620060756d2854abb79
paper_authors.name: Tyler Poon
paper_authors._id: 6466d620060756d2854abb7a
paper_authors.name: Andrew Tao
paper_authors._id: 6466d620060756d2854abb7b
paper_authors.name: Bryan Catanzaro
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-21T20:35:39.255Z
paper_authors._id: 6311021788942700629e6247
paper_authors.avatarUrl: /avatars/e7adc1632b76e80e7e4a590033d1c20a.svg
paper_authors.fullname: Bryan Catanzaro
paper_authors.type: user
paper_authors.user: ctnzr
paper_authors._id: 6466d620060756d2854abb7c
paper_authors.name: David Jacobs
paper_authors._id: 6466d620060756d2854abb7d
paper_authors.name: Jia-Bin Huang
paper_authors._id: 6466d620060756d2854abb7e
paper_authors.name: Ming-Yu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-21T20:36:52.721Z
paper_authors._id: 62f049afdf4b93aad5c7f2d6
paper_authors.avatarUrl: /avatars/e272e58ad996733d7098e50248e5b57e.svg
paper_authors.fullname: Ming-Yu Liu
paper_authors.type: user
paper_authors.user: mingyuliutw
paper_authors._id: 6466d620060756d2854abb7f
paper_authors.name: Yogesh Balaji
paper_title: Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models
paper_summary: Despite tremendous progress in generating high-quality images using diffusion
  models, synthesizing a sequence of animated frames that are both photorealistic
  and temporally coherent is still in its infancy. While off-the-shelf
  billion-scale datasets for image generation are available, collecting similar
  video data of the same scale is still challenging. Also, training a video
  diffusion model is computationally much more expensive than its image
  counterpart. In this work, we explore finetuning a pretrained image diffusion
  model with video data as a practical solution for the video synthesis task. We
  find that naively extending the image noise prior to video noise prior in video
  diffusion leads to sub-optimal performance. Our carefully designed video noise
  prior leads to substantially better performance. Extensive experimental
  validation shows that our model, Preserve Your Own Correlation (PYoCo), attains
  SOTA zero-shot text-to-video results on the UCF-101 and MSR-VTT benchmarks. It
  also achieves SOTA video generation quality on the small-scale UCF-101
  benchmark with a 10times smaller model using significantly less computation
  than the prior art.
paper_ai_summary: A pretrained image diffusion model fine-tuned with video data achieves state-of-the-art video synthesis quality and zero-shot text-to-video results with reduced computational cost.
paper_ai_keywords: diffusion models video noise prior image noise prior zero-shot text-to-video video generation quality UCF-101 MSR-VTT model fine-tuning