title: Factorized-Dreamer: Training A High-Quality Video Generator with Limited
    and Low-Quality Data
summary: Text-to-video (T2V) generation has gained significant attention due to its
  wide applications to video generation, editing, enhancement and translation,
  \etc. However, high-quality (HQ) video synthesis is extremely challenging
  because of the diverse and complex motions existed in real world. Most existing
  works struggle to address this problem by collecting large-scale HQ videos,
  which are inaccessible to the community. In this work, we show that publicly
  available limited and low-quality (LQ) data are sufficient to train a HQ video
  generator without recaptioning or finetuning. We factorize the whole T2V
  generation process into two steps: generating an image conditioned on a highly
  descriptive caption, and synthesizing the video conditioned on the generated
  image and a concise caption of motion details. Specifically, we present
  Factorized-Dreamer, a factorized spatiotemporal framework with several
  critical designs for T2V generation, including an adapter to combine text and
  image embeddings, a pixel-aware cross attention module to capture pixel-level
  image information, a T5 text encoder to better understand motion description,
  and a PredictNet to supervise optical flows. We further present a noise
  schedule, which plays a key role in ensuring the quality and stability of video
  generation. Our model lowers the requirements in detailed captions and HQ
  videos, and can be directly trained on limited LQ datasets with noisy and brief
  captions such as WebVid-10M, largely alleviating the cost to collect
  large-scale HQ video-text pairs. Extensive experiments in a variety of T2V and
  image-to-video generation tasks demonstrate the effectiveness of our proposed
  Factorized-Dreamer. Our source codes are available at
  https://github.com/yangxy/Factorized-Dreamer/.
paper_authors._id: 66c402084511077b9fca747e
paper_authors.name: Tao Yang
paper_authors._id: 66c402084511077b9fca747f
paper_authors.name: Yangming Shi
paper_authors._id: 66c402084511077b9fca7480
paper_authors.name: Yunwen Huang
paper_authors._id: 66c402084511077b9fca7481
paper_authors.name: Feng Chen
paper_authors._id: 66c402084511077b9fca7482
paper_authors.name: Yin Zheng
paper_authors._id: 66c402084511077b9fca7483
paper_authors.name: Lei Zhang
paper_title: Factorized-Dreamer: Training A High-Quality Video Generator with Limited
    and Low-Quality Data
paper_summary: Text-to-video (T2V) generation has gained significant attention due to its
  wide applications to video generation, editing, enhancement and translation,
  \etc. However, high-quality (HQ) video synthesis is extremely challenging
  because of the diverse and complex motions existed in real world. Most existing
  works struggle to address this problem by collecting large-scale HQ videos,
  which are inaccessible to the community. In this work, we show that publicly
  available limited and low-quality (LQ) data are sufficient to train a HQ video
  generator without recaptioning or finetuning. We factorize the whole T2V
  generation process into two steps: generating an image conditioned on a highly
  descriptive caption, and synthesizing the video conditioned on the generated
  image and a concise caption of motion details. Specifically, we present
  Factorized-Dreamer, a factorized spatiotemporal framework with several
  critical designs for T2V generation, including an adapter to combine text and
  image embeddings, a pixel-aware cross attention module to capture pixel-level
  image information, a T5 text encoder to better understand motion description,
  and a PredictNet to supervise optical flows. We further present a noise
  schedule, which plays a key role in ensuring the quality and stability of video
  generation. Our model lowers the requirements in detailed captions and HQ
  videos, and can be directly trained on limited LQ datasets with noisy and brief
  captions such as WebVid-10M, largely alleviating the cost to collect
  large-scale HQ video-text pairs. Extensive experiments in a variety of T2V and
  image-to-video generation tasks demonstrate the effectiveness of our proposed
  Factorized-Dreamer. Our source codes are available at
  https://github.com/yangxy/Factorized-Dreamer/.
paper_ai_summary: A factorized spatiotemporal framework, Factorized-Dreamer, enables high-quality video generation from limited and low-quality video data and concise captions.
paper_ai_keywords: Factorized-Dreamer spatiotemporal framework adapter pixel-aware cross attention module T5 text encoder PredictNet optical flows noise schedule WebVid-10M text-to-video generation image-to-video generation