UniVG: Towards UNIfied-modal Video Generation
Diffusion based video generation has received extensive attention and
achieved considerable success within both the academic and industrial
communities. However, current efforts are mainly concentrated on
single-objective or single-task video generation, such as generation driven by
text, by image, or by a combination of text and image. This cannot fully meet
the needs of real-world application scenarios, as users are likely to input
images and text conditions in a flexible manner, either individually or in
combination. To address this, we propose a Unified-modal Video Genearation
system that is capable of handling multiple video generation tasks across text
and image modalities. To this end, we revisit the various video generation
tasks within our system from the perspective of generative freedom, and
classify them into high-freedom and low-freedom video generation categories.
For high-freedom video generation, we employ Multi-condition Cross Attention to
generate videos that align with the semantics of the input images or text. For
low-freedom video generation, we introduce Biased Gaussian Noise to replace the
pure random Gaussian Noise, which helps to better preserve the content of the
input conditions. Our method achieves the lowest Fr\'echet Video Distance (FVD)
on the public academic benchmark MSR-VTT, surpasses the current open-source
methods in human evaluations, and is on par with the current close-source
method Gen2. For more samples, visit https://univg-baidu.github.io.
65a89543c5bf09bc9366b0ba
Ludan Ruan
admin_assigned
2024-01-18T09:14:23.454Z
64b6245c73b823a2c2998e78
/avatars/68cbbf267ee4f91302d0139739633c74.svg
Ludan Ruan
user
ludanruan
65a89543c5bf09bc9366b0bb
Lei Tian
65a89543c5bf09bc9366b0bc
Chuanwei Huang
65a89543c5bf09bc9366b0bd
Xu Zhang
65a89543c5bf09bc9366b0be
Xinyan Xiao
UniVG: Towards UNIfied-modal Video Generation
Diffusion based video generation has received extensive attention and
achieved considerable success within both the academic and industrial
communities. However, current efforts are mainly concentrated on
single-objective or single-task video generation, such as generation driven by
text, by image, or by a combination of text and image. This cannot fully meet
the needs of real-world application scenarios, as users are likely to input
images and text conditions in a flexible manner, either individually or in
combination. To address this, we propose a Unified-modal Video Genearation
system that is capable of handling multiple video generation tasks across text
and image modalities. To this end, we revisit the various video generation
tasks within our system from the perspective of generative freedom, and
classify them into high-freedom and low-freedom video generation categories.
For high-freedom video generation, we employ Multi-condition Cross Attention to
generate videos that align with the semantics of the input images or text. For
low-freedom video generation, we introduce Biased Gaussian Noise to replace the
pure random Gaussian Noise, which helps to better preserve the content of the
input conditions. Our method achieves the lowest Fr\'echet Video Distance (FVD)
on the public academic benchmark MSR-VTT, surpasses the current open-source
methods in human evaluations, and is on par with the current close-source
method Gen2. For more samples, visit https://univg-baidu.github.io.
The Unified-modal Video Generation system uses Multi-condition Cross Attention and Biased Gaussian Noise to handle diverse video generation tasks across text and image modalities, achieving superior performance on benchmarks.
diffusion based video generation Unified-modal Video Generation generative freedom Multi-condition Cross Attention Biased Gaussian Noise Fr\'echet Video Distance (FVD) MSR-VTT Gen2