VideoFactory: Swap Attention in Spatiotemporal Diffusions for
  Text-to-Video Generation
We present VideoFactory, an innovative framework for generating high-quality
open-domain videos. VideoFactory excels in producing high-definition
(1376x768), widescreen (16:9) videos without watermarks, creating an engaging
user experience. Generating videos guided by text instructions poses
significant challenges, such as modeling the complex relationship between space
and time, and the lack of large-scale text-video paired data. Previous
approaches extend pretrained text-to-image generation models by adding temporal
1D convolution/attention modules for video generation. However, these
approaches overlook the importance of jointly modeling space and time,
inevitably leading to temporal distortions and misalignment between texts and
videos. In this paper, we propose a novel approach that strengthens the
interaction between spatial and temporal perceptions. In particular, we utilize
a swapped cross-attention mechanism in 3D windows that alternates the "query"
role between spatial and temporal blocks, enabling mutual reinforcement for
each other. To fully unlock model capabilities for high-quality video
generation, we curate a large-scale video dataset called HD-VG-130M. This
dataset comprises 130 million text-video pairs from the open-domain, ensuring
high-definition, widescreen and watermark-free characters. Objective metrics
and user studies demonstrate the superiority of our approach in terms of
per-frame quality, temporal correlation, and text-video alignment, with clear
margins.
6466dcff5993aa7666cb791b
Wenjing Wang
admin_assigned
2023-05-21T20:30:40.197Z
637e0eacf09bf2498c53c757
/avatars/9694284767c63fbd11db1276711ccb50.svg
Wenjing Wang
user
daooshee
6466dcff5993aa7666cb791c
Huan Yang
admin_assigned
2023-05-21T20:32:04.654Z
639005861c5a6237272557d1
https://cdn-avatars.huggingface.co/v1/production/uploads/1670382915881-noauth.jpeg
Huan Yang
user
hyang0511
6466dcff5993aa7666cb791d
Zixi Tuo
6466dcff5993aa7666cb791e
Huiguo He
6466dcff5993aa7666cb791f
Junchen Zhu
claimed_verified
2024-05-13T07:51:37.821Z
6487e41bce70ef5e99e89b38
/avatars/35e468e76d8e0974f4b52844f44c5fda.svg
Junchen Zhu
user
Roseroll
6466dcff5993aa7666cb7920
Jianlong Fu
6466dcff5993aa7666cb7921
Jiaying Liu
VideoFactory: Swap Attention in Spatiotemporal Diffusions for
  Text-to-Video Generation
We present VideoFactory, an innovative framework for generating high-quality
open-domain videos. VideoFactory excels in producing high-definition
(1376x768), widescreen (16:9) videos without watermarks, creating an engaging
user experience. Generating videos guided by text instructions poses
significant challenges, such as modeling the complex relationship between space
and time, and the lack of large-scale text-video paired data. Previous
approaches extend pretrained text-to-image generation models by adding temporal
1D convolution/attention modules for video generation. However, these
approaches overlook the importance of jointly modeling space and time,
inevitably leading to temporal distortions and misalignment between texts and
videos. In this paper, we propose a novel approach that strengthens the
interaction between spatial and temporal perceptions. In particular, we utilize
a swapped cross-attention mechanism in 3D windows that alternates the "query"
role between spatial and temporal blocks, enabling mutual reinforcement for
each other. To fully unlock model capabilities for high-quality video
generation, we curate a large-scale video dataset called HD-VG-130M. This
dataset comprises 130 million text-video pairs from the open-domain, ensuring
high-definition, widescreen and watermark-free characters. Objective metrics
and user studies demonstrate the superiority of our approach in terms of
per-frame quality, temporal correlation, and text-video alignment, with clear
margins.
A new framework, VideoFactory, generates high-quality open-domain videos using a swapped cross-attention mechanism in 3D windows to improve spatial and temporal alignment, supported by a large-scale dataset.
text-to-image generation models temporal 1D convolution/attention modules swapped cross-attention mechanism 3D windows HD-VG-130M per-frame quality temporal correlation text-video alignment