title: A Recipe for Scaling up Text-to-Video Generation with Text-free Videos
summary: Diffusion-based text-to-video generation has witnessed impressive progress in
  the past year yet still falls behind text-to-image generation. One of the key
  reasons is the limited scale of publicly available data (e.g., 10M video-text
  pairs in WebVid10M vs. 5B image-text pairs in LAION), considering the high cost
  of video captioning. Instead, it could be far easier to collect unlabeled clips
  from video platforms like YouTube. Motivated by this, we come up with a novel
  text-to-video generation framework, termed TF-T2V, which can directly learn
  with text-free videos. The rationale behind is to separate the process of text
  decoding from that of temporal modeling. To this end, we employ a content
  branch and a motion branch, which are jointly optimized with weights shared.
  Following such a pipeline, we study the effect of doubling the scale of
  training set (i.e., video-only WebVid10M) with some randomly collected
  text-free videos and are encouraged to observe the performance improvement (FID
  from 9.67 to 8.19 and FVD from 484 to 441), demonstrating the scalability of
  our approach. We also find that our model could enjoy sustainable performance
  gain (FID from 8.19 to 7.64 and FVD from 441 to 366) after reintroducing some
  text labels for training. Finally, we validate the effectiveness and
  generalizability of our ideology on both native text-to-video generation and
  compositional video synthesis paradigms. Code and models will be publicly
  available at https://tf-t2v.github.io/.
paper_authors._id: 658ba1b801845f40cebc6e8e
paper_authors.name: Xiang Wang
paper_authors._id: 658ba1b801845f40cebc6e8f
paper_authors.name: Shiwei Zhang
paper_authors._id: 658ba1b801845f40cebc6e90
paper_authors.name: Hangjie Yuan
paper_authors._id: 658ba1b801845f40cebc6e91
paper_authors.name: Zhiwu Qing
paper_authors._id: 658ba1b801845f40cebc6e92
paper_authors.name: Biao Gong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T09:12:11.404Z
paper_authors._id: 644fcbea4f7316588267dc80
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/644fcbea4f7316588267dc80/w8-2Gkaw9BN9VzppNXrTP.jpeg
paper_authors.fullname: Biao Gong
paper_authors.type: user
paper_authors.user: BiaoGong
paper_authors._id: 658ba1b801845f40cebc6e93
paper_authors.name: Yingya Zhang
paper_authors._id: 658ba1b801845f40cebc6e94
paper_authors.name: Yujun Shen
paper_authors._id: 658ba1b801845f40cebc6e95
paper_authors.name: Changxin Gao
paper_authors._id: 658ba1b801845f40cebc6e96
paper_authors.name: Nong Sang
paper_title: A Recipe for Scaling up Text-to-Video Generation with Text-free Videos
paper_summary: Diffusion-based text-to-video generation has witnessed impressive progress in
  the past year yet still falls behind text-to-image generation. One of the key
  reasons is the limited scale of publicly available data (e.g., 10M video-text
  pairs in WebVid10M vs. 5B image-text pairs in LAION), considering the high cost
  of video captioning. Instead, it could be far easier to collect unlabeled clips
  from video platforms like YouTube. Motivated by this, we come up with a novel
  text-to-video generation framework, termed TF-T2V, which can directly learn
  with text-free videos. The rationale behind is to separate the process of text
  decoding from that of temporal modeling. To this end, we employ a content
  branch and a motion branch, which are jointly optimized with weights shared.
  Following such a pipeline, we study the effect of doubling the scale of
  training set (i.e., video-only WebVid10M) with some randomly collected
  text-free videos and are encouraged to observe the performance improvement (FID
  from 9.67 to 8.19 and FVD from 484 to 441), demonstrating the scalability of
  our approach. We also find that our model could enjoy sustainable performance
  gain (FID from 8.19 to 7.64 and FVD from 441 to 366) after reintroducing some
  text labels for training. Finally, we validate the effectiveness and
  generalizability of our ideology on both native text-to-video generation and
  compositional video synthesis paradigms. Code and models will be publicly
  available at https://tf-t2v.github.io/.
paper_ai_keywords: diffusion-based text-to-video generation text-to-image generation video-text pairs WebVid10M image-text pairs LAION video captioning text-free videos YouTube TF-T2V text decoding temporal modeling content branch motion branch FID FVD video-only WebVid10M compositional video synthesis