title: LayerFlow: A Unified Model for Layer-aware Video Generation
summary: We present LayerFlow, a unified solution for layer-aware video generation.
  Given per-layer prompts, LayerFlow generates videos for the transparent
  foreground, clean background, and blended scene. It also supports versatile
  variants like decomposing a blended video or generating the background for the
  given foreground and vice versa. Starting from a text-to-video diffusion
  transformer, we organize the videos for different layers as sub-clips, and
  leverage layer embeddings to distinguish each clip and the corresponding
  layer-wise prompts. In this way, we seamlessly support the aforementioned
  variants in one unified framework. For the lack of high-quality layer-wise
  training videos, we design a multi-stage training strategy to accommodate
  static images with high-quality layer annotations. Specifically, we first train
  the model with low-quality video data. Then, we tune a motion LoRA to make the
  model compatible with static frames. Afterward, we train the content LoRA on
  the mixture of image data with high-quality layered images along with
  copy-pasted video data. During inference, we remove the motion LoRA thus
  generating smooth videos with desired layers.
paper_authors._id: 684103aed45a1fc5540ddc10
paper_authors.name: Sihui Ji
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T12:48:05.938Z
paper_authors._id: 680c7488195bc1af6f557b05
paper_authors.avatarUrl: /avatars/e6c5a88023ab167c777a6e8915c560a2.svg
paper_authors.fullname: Sihui Ji
paper_authors.type: user
paper_authors.user: Jish2
paper_authors._id: 684103aed45a1fc5540ddc11
paper_authors.name: Hao Luo
paper_authors._id: 684103aed45a1fc5540ddc12
paper_authors.name: Xi Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-05T07:26:50.027Z
paper_authors._id: 644a1b6401e18bf93a6f45c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/644a1b6401e18bf93a6f45c1/P0i_CgCrIzOS2tYRlxoE9.png
paper_authors.fullname: xichen
paper_authors.type: user
paper_authors.user: xichenhku
paper_authors._id: 684103aed45a1fc5540ddc13
paper_authors.name: Yuanpeng Tu
paper_authors._id: 684103aed45a1fc5540ddc14
paper_authors.name: Yiyang Wang
paper_authors._id: 684103aed45a1fc5540ddc15
paper_authors.name: Hengshuang Zhao
paper_title: LayerFlow: A Unified Model for Layer-aware Video Generation
paper_summary: We present LayerFlow, a unified solution for layer-aware video generation.
  Given per-layer prompts, LayerFlow generates videos for the transparent
  foreground, clean background, and blended scene. It also supports versatile
  variants like decomposing a blended video or generating the background for the
  given foreground and vice versa. Starting from a text-to-video diffusion
  transformer, we organize the videos for different layers as sub-clips, and
  leverage layer embeddings to distinguish each clip and the corresponding
  layer-wise prompts. In this way, we seamlessly support the aforementioned
  variants in one unified framework. For the lack of high-quality layer-wise
  training videos, we design a multi-stage training strategy to accommodate
  static images with high-quality layer annotations. Specifically, we first train
  the model with low-quality video data. Then, we tune a motion LoRA to make the
  model compatible with static frames. Afterward, we train the content LoRA on
  the mixture of image data with high-quality layered images along with
  copy-pasted video data. During inference, we remove the motion LoRA thus
  generating smooth videos with desired layers.
paper_ai_summary: LayerFlow is a unified framework for generating layer-aware videos using a text-to-video diffusion transformer and layer embeddings, supporting various video generation tasks with a multi-stage training strategy.
paper_ai_keywords: LayerFlow text-to-video diffusion transformer layer embeddings sub-clips multi-stage training strategy motion LoRA content LoRA layered images copy-pasted video data smooth videos