title: VideoFactory: Swap Attention in Spatiotemporal Diffusions for
    Text-to-Video Generation
summary: We present VideoFactory, an innovative framework for generating high-quality
  open-domain videos. VideoFactory excels in producing high-definition
  (1376x768), widescreen (16:9) videos without watermarks, creating an engaging
  user experience. Generating videos guided by text instructions poses
  significant challenges, such as modeling the complex relationship between space
  and time, and the lack of large-scale text-video paired data. Previous
  approaches extend pretrained text-to-image generation models by adding temporal
  1D convolution/attention modules for video generation. However, these
  approaches overlook the importance of jointly modeling space and time,
  inevitably leading to temporal distortions and misalignment between texts and
  videos. In this paper, we propose a novel approach that strengthens the
  interaction between spatial and temporal perceptions. In particular, we utilize
  a swapped cross-attention mechanism in 3D windows that alternates the "query"
  role between spatial and temporal blocks, enabling mutual reinforcement for
  each other. To fully unlock model capabilities for high-quality video
  generation, we curate a large-scale video dataset called HD-VG-130M. This
  dataset comprises 130 million text-video pairs from the open-domain, ensuring
  high-definition, widescreen and watermark-free characters. Objective metrics
  and user studies demonstrate the superiority of our approach in terms of
  per-frame quality, temporal correlation, and text-video alignment, with clear
  margins.
paper_authors._id: 6466dcff5993aa7666cb791b
paper_authors.name: Wenjing Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-21T20:30:40.197Z
paper_authors._id: 637e0eacf09bf2498c53c757
paper_authors.avatarUrl: /avatars/9694284767c63fbd11db1276711ccb50.svg
paper_authors.fullname: Wenjing Wang
paper_authors.type: user
paper_authors.user: daooshee
paper_authors._id: 6466dcff5993aa7666cb791c
paper_authors.name: Huan Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-21T20:32:04.654Z
paper_authors._id: 639005861c5a6237272557d1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1670382915881-noauth.jpeg
paper_authors.fullname: Huan Yang
paper_authors.type: user
paper_authors.user: hyang0511
paper_authors._id: 6466dcff5993aa7666cb791d
paper_authors.name: Zixi Tuo
paper_authors._id: 6466dcff5993aa7666cb791e
paper_authors.name: Huiguo He
paper_authors._id: 6466dcff5993aa7666cb791f
paper_authors.name: Junchen Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-13T07:51:37.821Z
paper_authors._id: 6487e41bce70ef5e99e89b38
paper_authors.avatarUrl: /avatars/35e468e76d8e0974f4b52844f44c5fda.svg
paper_authors.fullname: Junchen Zhu
paper_authors.type: user
paper_authors.user: Roseroll
paper_authors._id: 6466dcff5993aa7666cb7920
paper_authors.name: Jianlong Fu
paper_authors._id: 6466dcff5993aa7666cb7921
paper_authors.name: Jiaying Liu
paper_title: VideoFactory: Swap Attention in Spatiotemporal Diffusions for
    Text-to-Video Generation
paper_summary: We present VideoFactory, an innovative framework for generating high-quality
  open-domain videos. VideoFactory excels in producing high-definition
  (1376x768), widescreen (16:9) videos without watermarks, creating an engaging
  user experience. Generating videos guided by text instructions poses
  significant challenges, such as modeling the complex relationship between space
  and time, and the lack of large-scale text-video paired data. Previous
  approaches extend pretrained text-to-image generation models by adding temporal
  1D convolution/attention modules for video generation. However, these
  approaches overlook the importance of jointly modeling space and time,
  inevitably leading to temporal distortions and misalignment between texts and
  videos. In this paper, we propose a novel approach that strengthens the
  interaction between spatial and temporal perceptions. In particular, we utilize
  a swapped cross-attention mechanism in 3D windows that alternates the "query"
  role between spatial and temporal blocks, enabling mutual reinforcement for
  each other. To fully unlock model capabilities for high-quality video
  generation, we curate a large-scale video dataset called HD-VG-130M. This
  dataset comprises 130 million text-video pairs from the open-domain, ensuring
  high-definition, widescreen and watermark-free characters. Objective metrics
  and user studies demonstrate the superiority of our approach in terms of
  per-frame quality, temporal correlation, and text-video alignment, with clear
  margins.
paper_ai_keywords: cross-attention actors 3D temporal convolution video dataset HD-VG-130M per-frame quality temporal correlation