title: xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed
    Representations
summary: We present xGen-VideoSyn-1, a text-to-video (T2V) generation model capable of
  producing realistic scenes from textual descriptions. Building on recent
  advancements, such as OpenAI's Sora, we explore the latent diffusion model
  (LDM) architecture and introduce a video variational autoencoder (VidVAE).
  VidVAE compresses video data both spatially and temporally, significantly
  reducing the length of visual tokens and the computational demands associated
  with generating long-sequence videos. To further address the computational
  costs, we propose a divide-and-merge strategy that maintains temporal
  consistency across video segments. Our Diffusion Transformer (DiT) model
  incorporates spatial and temporal self-attention layers, enabling robust
  generalization across different timeframes and aspect ratios. We have devised a
  data processing pipeline from the very beginning and collected over 13M
  high-quality video-text pairs. The pipeline includes multiple steps such as
  clipping, text detection, motion estimation, aesthetics scoring, and dense
  captioning based on our in-house video-LLM model. Training the VidVAE and DiT
  models required approximately 40 and 642 H100 days, respectively. Our model
  supports over 14-second 720p video generation in an end-to-end way and
  demonstrates competitive performance against state-of-the-art T2V models.
paper_authors._id: 66c7ef2596661a3c4eaa9fb5
paper_authors.name: Can Qin
paper_authors._id: 66c7ef2596661a3c4eaa9fb6
paper_authors.name: Congying Xia
paper_authors._id: 66c7ef2596661a3c4eaa9fb7
paper_authors.name: Krithika Ramakrishnan
paper_authors._id: 66c7ef2596661a3c4eaa9fb8
paper_authors.name: Michael Ryoo
paper_authors._id: 66c7ef2596661a3c4eaa9fb9
paper_authors.name: Lifu Tu
paper_authors._id: 66c7ef2596661a3c4eaa9fba
paper_authors.name: Yihao Feng
paper_authors._id: 66c7ef2596661a3c4eaa9fbb
paper_authors.name: Manli Shu
paper_authors._id: 66c7ef2596661a3c4eaa9fbc
paper_authors.name: Honglu Zhou
paper_authors._id: 66c7ef2596661a3c4eaa9fbd
paper_authors.name: Anas Awadalla
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-29T07:23:43.580Z
paper_authors._id: 60f1ab3afe0d78e01037eeb1
paper_authors.avatarUrl: /avatars/f784fa423fd84fffb4683fa837ffc5a3.svg
paper_authors.fullname: Anas Awadalla
paper_authors.type: user
paper_authors.user: anas-awadalla
paper_authors._id: 66c7ef2596661a3c4eaa9fbe
paper_authors.name: Jun Wang
paper_authors._id: 66c7ef2596661a3c4eaa9fbf
paper_authors.name: Senthil Purushwalkam
paper_authors._id: 66c7ef2596661a3c4eaa9fc0
paper_authors.name: Le Xue
paper_authors._id: 66c7ef2596661a3c4eaa9fc1
paper_authors.name: Yingbo Zhou
paper_authors._id: 66c7ef2596661a3c4eaa9fc2
paper_authors.name: Huan Wang
paper_authors._id: 66c7ef2596661a3c4eaa9fc3
paper_authors.name: Silvio Savarese
paper_authors._id: 66c7ef2596661a3c4eaa9fc4
paper_authors.name: Juan Carlos Niebles
paper_authors._id: 66c7ef2596661a3c4eaa9fc5
paper_authors.name: Zeyuan Chen
paper_authors._id: 66c7ef2596661a3c4eaa9fc6
paper_authors.name: Ran Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-15T10:41:43.287Z
paper_authors._id: 6465c4c863e7e09dd02e3e1b
paper_authors.avatarUrl: /avatars/200b029184d2616f98296a2c212f0785.svg
paper_authors.fullname: Ran Xu
paper_authors.type: user
paper_authors.user: xurantju
paper_authors._id: 66c7ef2596661a3c4eaa9fc7
paper_authors.name: Caiming Xiong
paper_title: xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed
    Representations
paper_summary: We present xGen-VideoSyn-1, a text-to-video (T2V) generation model capable of
  producing realistic scenes from textual descriptions. Building on recent
  advancements, such as OpenAI's Sora, we explore the latent diffusion model
  (LDM) architecture and introduce a video variational autoencoder (VidVAE).
  VidVAE compresses video data both spatially and temporally, significantly
  reducing the length of visual tokens and the computational demands associated
  with generating long-sequence videos. To further address the computational
  costs, we propose a divide-and-merge strategy that maintains temporal
  consistency across video segments. Our Diffusion Transformer (DiT) model
  incorporates spatial and temporal self-attention layers, enabling robust
  generalization across different timeframes and aspect ratios. We have devised a
  data processing pipeline from the very beginning and collected over 13M
  high-quality video-text pairs. The pipeline includes multiple steps such as
  clipping, text detection, motion estimation, aesthetics scoring, and dense
  captioning based on our in-house video-LLM model. Training the VidVAE and DiT
  models required approximately 40 and 642 H100 days, respectively. Our model
  supports over 14-second 720p video generation in an end-to-end way and
  demonstrates competitive performance against state-of-the-art T2V models.
paper_ai_summary: The xGen-VideoSyn-1 model generates realistic videos from text using a Latent Diffusion Model and Video Variational Autoencoder, with a spatial and temporal divide-and-merge strategy and Diffusion Transformer.
paper_ai_keywords: latent diffusion model VVVAE spatial compression temporal compression visual tokens computational demands divide-and-merge strategy temporal consistency Diffusion Transformer spatial self-attention temporal self-attention robust generalization text-to-video video-text pairs clippng text detection motion estimation aesthetics scoring dense captioning video-LLM model