title: UniVG: Towards UNIfied-modal Video Generation
summary: Diffusion based video generation has received extensive attention and
  achieved considerable success within both the academic and industrial
  communities. However, current efforts are mainly concentrated on
  single-objective or single-task video generation, such as generation driven by
  text, by image, or by a combination of text and image. This cannot fully meet
  the needs of real-world application scenarios, as users are likely to input
  images and text conditions in a flexible manner, either individually or in
  combination. To address this, we propose a Unified-modal Video Genearation
  system that is capable of handling multiple video generation tasks across text
  and image modalities. To this end, we revisit the various video generation
  tasks within our system from the perspective of generative freedom, and
  classify them into high-freedom and low-freedom video generation categories.
  For high-freedom video generation, we employ Multi-condition Cross Attention to
  generate videos that align with the semantics of the input images or text. For
  low-freedom video generation, we introduce Biased Gaussian Noise to replace the
  pure random Gaussian Noise, which helps to better preserve the content of the
  input conditions. Our method achieves the lowest Fr\'echet Video Distance (FVD)
  on the public academic benchmark MSR-VTT, surpasses the current open-source
  methods in human evaluations, and is on par with the current close-source
  method Gen2. For more samples, visit https://univg-baidu.github.io.
paper_authors._id: 65a89543c5bf09bc9366b0ba
paper_authors.name: Ludan Ruan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-18T09:14:23.454Z
paper_authors._id: 64b6245c73b823a2c2998e78
paper_authors.avatarUrl: /avatars/68cbbf267ee4f91302d0139739633c74.svg
paper_authors.fullname: Ludan Ruan
paper_authors.type: user
paper_authors.user: ludanruan
paper_authors._id: 65a89543c5bf09bc9366b0bb
paper_authors.name: Lei Tian
paper_authors._id: 65a89543c5bf09bc9366b0bc
paper_authors.name: Chuanwei Huang
paper_authors._id: 65a89543c5bf09bc9366b0bd
paper_authors.name: Xu Zhang
paper_authors._id: 65a89543c5bf09bc9366b0be
paper_authors.name: Xinyan Xiao
paper_title: UniVG: Towards UNIfied-modal Video Generation
paper_summary: Diffusion based video generation has received extensive attention and
  achieved considerable success within both the academic and industrial
  communities. However, current efforts are mainly concentrated on
  single-objective or single-task video generation, such as generation driven by
  text, by image, or by a combination of text and image. This cannot fully meet
  the needs of real-world application scenarios, as users are likely to input
  images and text conditions in a flexible manner, either individually or in
  combination. To address this, we propose a Unified-modal Video Genearation
  system that is capable of handling multiple video generation tasks across text
  and image modalities. To this end, we revisit the various video generation
  tasks within our system from the perspective of generative freedom, and
  classify them into high-freedom and low-freedom video generation categories.
  For high-freedom video generation, we employ Multi-condition Cross Attention to
  generate videos that align with the semantics of the input images or text. For
  low-freedom video generation, we introduce Biased Gaussian Noise to replace the
  pure random Gaussian Noise, which helps to better preserve the content of the
  input conditions. Our method achieves the lowest Fr\'echet Video Distance (FVD)
  on the public academic benchmark MSR-VTT, surpasses the current open-source
  methods in human evaluations, and is on par with the current close-source
  method Gen2. For more samples, visit https://univg-baidu.github.io.
paper_ai_keywords: diffusion based video generation video generation cross attention generative freedom high-freedom video generation low-freedom video generation Multi-condition Cross Attention Biased Gaussian Noise Gaussian Noise Fr\'echet Video Distance (FVD) MSR-VTT