title: VideoGen: A Reference-Guided Latent Diffusion Approach for High
    Definition Text-to-Video Generation
summary: In this paper, we present VideoGen, a text-to-video generation approach,
  which can generate a high-definition video with high frame fidelity and strong
  temporal consistency using reference-guided latent diffusion. We leverage an
  off-the-shelf text-to-image generation model, e.g., Stable Diffusion, to
  generate an image with high content quality from the text prompt, as a
  reference image to guide video generation. Then, we introduce an efficient
  cascaded latent diffusion module conditioned on both the reference image and
  the text prompt, for generating latent video representations, followed by a
  flow-based temporal upsampling step to improve the temporal resolution.
  Finally, we map latent video representations into a high-definition video
  through an enhanced video decoder. During training, we use the first frame of a
  ground-truth video as the reference image for training the cascaded latent
  diffusion module. The main characterises of our approach include: the reference
  image generated by the text-to-image model improves the visual fidelity; using
  it as the condition makes the diffusion model focus more on learning the video
  dynamics; and the video decoder is trained over unlabeled video data, thus
  benefiting from high-quality easily-available videos. VideoGen sets a new
  state-of-the-art in text-to-video generation in terms of both qualitative and
  quantitative evaluation.
paper_authors._id: 64f52ebb4dc8a69ca0155a6e
paper_authors.name: Xin Li
paper_authors._id: 64f52ebb4dc8a69ca0155a6f
paper_authors.name: Wenqing Chu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-04T15:05:35.638Z
paper_authors._id: 634e084dc1ce28f1de916787
paper_authors.avatarUrl: /avatars/f233256af736bbc7d4d7e92df4b04e98.svg
paper_authors.fullname: wenqing chu
paper_authors.type: user
paper_authors.user: wenqingchu
paper_authors._id: 64f52ebb4dc8a69ca0155a70
paper_authors.name: Ye Wu
paper_authors._id: 64f52ebb4dc8a69ca0155a71
paper_authors.name: Weihang Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-04T08:56:47.360Z
paper_authors._id: 64c22c5dd1ca220e30433db4
paper_authors.avatarUrl: /avatars/7b0f3d87ea54686b88fa37fe0e542e29.svg
paper_authors.fullname: yuan
paper_authors.type: user
paper_authors.user: weihangY
paper_authors._id: 64f52ebb4dc8a69ca0155a72
paper_authors.name: Fanglong Liu
paper_authors._id: 64f52ebb4dc8a69ca0155a73
paper_authors.name: Qi Zhang
paper_authors._id: 64f52ebb4dc8a69ca0155a74
paper_authors.name: Fu Li
paper_authors._id: 64f52ebb4dc8a69ca0155a75
paper_authors.name: Haocheng Feng
paper_authors._id: 64f52ebb4dc8a69ca0155a76
paper_authors.name: Errui Ding
paper_authors._id: 64f52ebb4dc8a69ca0155a77
paper_authors.name: Jingdong Wang
paper_title: VideoGen: A Reference-Guided Latent Diffusion Approach for High
    Definition Text-to-Video Generation
paper_summary: In this paper, we present VideoGen, a text-to-video generation approach,
  which can generate a high-definition video with high frame fidelity and strong
  temporal consistency using reference-guided latent diffusion. We leverage an
  off-the-shelf text-to-image generation model, e.g., Stable Diffusion, to
  generate an image with high content quality from the text prompt, as a
  reference image to guide video generation. Then, we introduce an efficient
  cascaded latent diffusion module conditioned on both the reference image and
  the text prompt, for generating latent video representations, followed by a
  flow-based temporal upsampling step to improve the temporal resolution.
  Finally, we map latent video representations into a high-definition video
  through an enhanced video decoder. During training, we use the first frame of a
  ground-truth video as the reference image for training the cascaded latent
  diffusion module. The main characterises of our approach include: the reference
  image generated by the text-to-image model improves the visual fidelity; using
  it as the condition makes the diffusion model focus more on learning the video
  dynamics; and the video decoder is trained over unlabeled video data, thus
  benefiting from high-quality easily-available videos. VideoGen sets a new
  state-of-the-art in text-to-video generation in terms of both qualitative and
  quantitative evaluation.
paper_ai_summary: VideoGen uses text-to-image models and latent diffusion to generate high-definition, temporally consistent videos from text prompts, achieving state-of-the-art performance.
paper_ai_keywords: text-to-video generation VideoGen latent diffusion text-to-image generation reference-guided latent diffusion cascaded latent diffusion temporal upsampling video decoder video dynamics