title: Reuse and Diffuse: Iterative Denoising for Text-to-Video Generation
summary: Inspired by the remarkable success of Latent Diffusion Models (LDMs) for
  image synthesis, we study LDM for text-to-video generation, which is a
  formidable challenge due to the computational and memory constraints during
  both model training and inference. A single LDM is usually only capable of
  generating a very limited number of video frames. Some existing works focus on
  separate prediction models for generating more video frames, which suffer from
  additional training cost and frame-level jittering, however. In this paper, we
  propose a framework called "Reuse and Diffuse" dubbed VidRD to
  produce more frames following the frames already generated by an LDM.
  Conditioned on an initial video clip with a small number of frames, additional
  frames are iteratively generated by reusing the original latent features and
  following the previous diffusion process. Besides, for the autoencoder used for
  translation between pixel space and latent space, we inject temporal layers
  into its decoder and fine-tune these layers for higher temporal consistency. We
  also propose a set of strategies for composing video-text data that involve
  diverse content from multiple existing datasets including video datasets for
  action recognition and image-text datasets. Extensive experiments show that our
  method achieves good results in both quantitative and qualitative evaluations.
  Our project page is available
  https://anonymous0x233.github.io/ReuseAndDiffuse/{here}.
paper_authors._id: 64fa8ee9da4429f25dce324c
paper_authors.name: Jiaxi Gu
paper_authors._id: 64fa8ee9da4429f25dce324d
paper_authors.name: Shicong Wang
paper_authors._id: 64fa8ee9da4429f25dce324e
paper_authors.name: Haoyu Zhao
paper_authors._id: 64fa8ee9da4429f25dce324f
paper_authors.name: Tianyi Lu
paper_authors._id: 64fa8ee9da4429f25dce3250
paper_authors.name: Xing Zhang
paper_authors._id: 64fa8ee9da4429f25dce3251
paper_authors.name: Zuxuan Wu
paper_authors._id: 64fa8ee9da4429f25dce3252
paper_authors.name: Songcen Xu
paper_authors._id: 64fa8ee9da4429f25dce3253
paper_authors.name: Wei Zhang
paper_authors._id: 64fa8ee9da4429f25dce3254
paper_authors.name: Yu-Gang Jiang
paper_authors._id: 64fa8ee9da4429f25dce3255
paper_authors.name: Hang Xu
paper_title: Reuse and Diffuse: Iterative Denoising for Text-to-Video Generation
paper_summary: Inspired by the remarkable success of Latent Diffusion Models (LDMs) for
  image synthesis, we study LDM for text-to-video generation, which is a
  formidable challenge due to the computational and memory constraints during
  both model training and inference. A single LDM is usually only capable of
  generating a very limited number of video frames. Some existing works focus on
  separate prediction models for generating more video frames, which suffer from
  additional training cost and frame-level jittering, however. In this paper, we
  propose a framework called "Reuse and Diffuse" dubbed VidRD to
  produce more frames following the frames already generated by an LDM.
  Conditioned on an initial video clip with a small number of frames, additional
  frames are iteratively generated by reusing the original latent features and
  following the previous diffusion process. Besides, for the autoencoder used for
  translation between pixel space and latent space, we inject temporal layers
  into its decoder and fine-tune these layers for higher temporal consistency. We
  also propose a set of strategies for composing video-text data that involve
  diverse content from multiple existing datasets including video datasets for
  action recognition and image-text datasets. Extensive experiments show that our
  method achieves good results in both quantitative and qualitative evaluations.
  Our project page is available
  https://anonymous0x233.github.io/ReuseAndDiffuse/{here}.
paper_ai_summary: A Reuse and Diffuse framework (VidRD) enhances text-to-video generation by iteratively adding frames to initial video clips using latent features and temporal consistency enhancements.
paper_ai_keywords: Latent Diffusion Models (LDMs) text-to-video generation diffusion models latent space pixel space temporal layers autoencoder video-text data video datasets image-text datasets temporal consistency