LayerFlow: A Unified Model for Layer-aware Video Generation
We present LayerFlow, a unified solution for layer-aware video generation.
Given per-layer prompts, LayerFlow generates videos for the transparent
foreground, clean background, and blended scene. It also supports versatile
variants like decomposing a blended video or generating the background for the
given foreground and vice versa. Starting from a text-to-video diffusion
transformer, we organize the videos for different layers as sub-clips, and
leverage layer embeddings to distinguish each clip and the corresponding
layer-wise prompts. In this way, we seamlessly support the aforementioned
variants in one unified framework. For the lack of high-quality layer-wise
training videos, we design a multi-stage training strategy to accommodate
static images with high-quality layer annotations. Specifically, we first train
the model with low-quality video data. Then, we tune a motion LoRA to make the
model compatible with static frames. Afterward, we train the content LoRA on
the mixture of image data with high-quality layered images along with
copy-pasted video data. During inference, we remove the motion LoRA thus
generating smooth videos with desired layers.
684103aed45a1fc5540ddc10
Sihui Ji
admin_assigned
2025-06-05T12:48:05.938Z
680c7488195bc1af6f557b05
/avatars/e6c5a88023ab167c777a6e8915c560a2.svg
Sihui Ji
user
Jish2
684103aed45a1fc5540ddc11
Hao Luo
684103aed45a1fc5540ddc12
Xi Chen
claimed_verified
2025-06-05T07:26:50.027Z
644a1b6401e18bf93a6f45c1
https://cdn-avatars.huggingface.co/v1/production/uploads/644a1b6401e18bf93a6f45c1/P0i_CgCrIzOS2tYRlxoE9.png
xichen
user
xichenhku
684103aed45a1fc5540ddc13
Yuanpeng Tu
684103aed45a1fc5540ddc14
Yiyang Wang
684103aed45a1fc5540ddc15
Hengshuang Zhao
LayerFlow: A Unified Model for Layer-aware Video Generation
We present LayerFlow, a unified solution for layer-aware video generation.
Given per-layer prompts, LayerFlow generates videos for the transparent
foreground, clean background, and blended scene. It also supports versatile
variants like decomposing a blended video or generating the background for the
given foreground and vice versa. Starting from a text-to-video diffusion
transformer, we organize the videos for different layers as sub-clips, and
leverage layer embeddings to distinguish each clip and the corresponding
layer-wise prompts. In this way, we seamlessly support the aforementioned
variants in one unified framework. For the lack of high-quality layer-wise
training videos, we design a multi-stage training strategy to accommodate
static images with high-quality layer annotations. Specifically, we first train
the model with low-quality video data. Then, we tune a motion LoRA to make the
model compatible with static frames. Afterward, we train the content LoRA on
the mixture of image data with high-quality layered images along with
copy-pasted video data. During inference, we remove the motion LoRA thus
generating smooth videos with desired layers.
LayerFlow is a unified framework for generating layer-aware videos using a text-to-video diffusion transformer and layer embeddings, supporting various video generation tasks with a multi-stage training strategy.
LayerFlow text-to-video diffusion transformer layer embeddings sub-clips multi-stage training strategy motion LoRA content LoRA layered images copy-pasted video data smooth videos