xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed
  Representations
We present xGen-VideoSyn-1, a text-to-video (T2V) generation model capable of
producing realistic scenes from textual descriptions. Building on recent
advancements, such as OpenAI's Sora, we explore the latent diffusion model
(LDM) architecture and introduce a video variational autoencoder (VidVAE).
VidVAE compresses video data both spatially and temporally, significantly
reducing the length of visual tokens and the computational demands associated
with generating long-sequence videos. To further address the computational
costs, we propose a divide-and-merge strategy that maintains temporal
consistency across video segments. Our Diffusion Transformer (DiT) model
incorporates spatial and temporal self-attention layers, enabling robust
generalization across different timeframes and aspect ratios. We have devised a
data processing pipeline from the very beginning and collected over 13M
high-quality video-text pairs. The pipeline includes multiple steps such as
clipping, text detection, motion estimation, aesthetics scoring, and dense
captioning based on our in-house video-LLM model. Training the VidVAE and DiT
models required approximately 40 and 642 H100 days, respectively. Our model
supports over 14-second 720p video generation in an end-to-end way and
demonstrates competitive performance against state-of-the-art T2V models.
66c7ef2596661a3c4eaa9fb5
Can Qin
66c7ef2596661a3c4eaa9fb6
Congying Xia
66c7ef2596661a3c4eaa9fb7
Krithika Ramakrishnan
66c7ef2596661a3c4eaa9fb8
Michael Ryoo
66c7ef2596661a3c4eaa9fb9
Lifu Tu
66c7ef2596661a3c4eaa9fba
Yihao Feng
66c7ef2596661a3c4eaa9fbb
Manli Shu
66c7ef2596661a3c4eaa9fbc
Honglu Zhou
66c7ef2596661a3c4eaa9fbd
Anas Awadalla
claimed_verified
2024-08-29T07:23:43.580Z
60f1ab3afe0d78e01037eeb1
/avatars/f784fa423fd84fffb4683fa837ffc5a3.svg
Anas Awadalla
user
anas-awadalla
66c7ef2596661a3c4eaa9fbe
Jun Wang
66c7ef2596661a3c4eaa9fbf
Senthil Purushwalkam
66c7ef2596661a3c4eaa9fc0
Le Xue
66c7ef2596661a3c4eaa9fc1
Yingbo Zhou
66c7ef2596661a3c4eaa9fc2
Huan Wang
66c7ef2596661a3c4eaa9fc3
Silvio Savarese
66c7ef2596661a3c4eaa9fc4
Juan Carlos Niebles
66c7ef2596661a3c4eaa9fc5
Zeyuan Chen
66c7ef2596661a3c4eaa9fc6
Ran Xu
claimed_verified
2025-05-15T10:41:43.287Z
6465c4c863e7e09dd02e3e1b
/avatars/200b029184d2616f98296a2c212f0785.svg
Ran Xu
user
xurantju
66c7ef2596661a3c4eaa9fc7
Caiming Xiong
xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed
  Representations
We present xGen-VideoSyn-1, a text-to-video (T2V) generation model capable of
producing realistic scenes from textual descriptions. Building on recent
advancements, such as OpenAI's Sora, we explore the latent diffusion model
(LDM) architecture and introduce a video variational autoencoder (VidVAE).
VidVAE compresses video data both spatially and temporally, significantly
reducing the length of visual tokens and the computational demands associated
with generating long-sequence videos. To further address the computational
costs, we propose a divide-and-merge strategy that maintains temporal
consistency across video segments. Our Diffusion Transformer (DiT) model
incorporates spatial and temporal self-attention layers, enabling robust
generalization across different timeframes and aspect ratios. We have devised a
data processing pipeline from the very beginning and collected over 13M
high-quality video-text pairs. The pipeline includes multiple steps such as
clipping, text detection, motion estimation, aesthetics scoring, and dense
captioning based on our in-house video-LLM model. Training the VidVAE and DiT
models required approximately 40 and 642 H100 days, respectively. Our model
supports over 14-second 720p video generation in an end-to-end way and
demonstrates competitive performance against state-of-the-art T2V models.
The xGen-VideoSyn-1 model generates realistic videos from text using a Latent Diffusion Model and Video Variational Autoencoder, with a spatial and temporal divide-and-merge strategy and Diffusion Transformer.
latent diffusion model VVVAE spatial compression temporal compression visual tokens computational demands divide-and-merge strategy temporal consistency Diffusion Transformer spatial self-attention temporal self-attention robust generalization text-to-video video-text pairs clippng text detection motion estimation aesthetics scoring dense captioning video-LLM model