VACE: All-in-One Video Creation and Editing
Diffusion Transformer has demonstrated powerful capability and scalability in
generating high-quality images and videos. Further pursuing the unification of
generation and editing tasks has yielded significant progress in the domain of
image content creation. However, due to the intrinsic demands for consistency
across both temporal and spatial dynamics, achieving a unified approach for
video synthesis remains challenging. We introduce VACE, which enables users to
perform Video tasks within an All-in-one framework for Creation and Editing.
These tasks include reference-to-video generation, video-to-video editing, and
masked video-to-video editing. Specifically, we effectively integrate the
requirements of various tasks by organizing video task inputs, such as editing,
reference, and masking, into a unified interface referred to as the Video
Condition Unit (VCU). Furthermore, by utilizing a Context Adapter structure, we
inject different task concepts into the model using formalized representations
of temporal and spatial dimensions, allowing it to handle arbitrary video
synthesis tasks flexibly. Extensive experiments demonstrate that the unified
model of VACE achieves performance on par with task-specific models across
various subtasks. Simultaneously, it enables diverse applications through
versatile task combinations. Project page:
https://ali-vilab.github.io/VACE-Page/.
67d04a5d7c367a40ae3e417a
Zeyinzi Jiang
claimed_verified
2025-03-12T08:42:12.022Z
643d278b482011f5f2bd0fae
/avatars/70b8a7ffbfa2a1c4b6f5ff5e2b96b7bf.svg
jiangzeyinzi
user
jiangzeyinzi
67d04a5d7c367a40ae3e417b
Zhen Han
claimed_verified
2025-03-12T08:42:18.152Z
647ffabf28b737d7b9462eb2
/avatars/210441fc6645d08b36ad43734108f914.svg
Zhen Han
user
hanzhn
67d04a5d7c367a40ae3e417c
Chaojie Mao
claimed_verified
2025-03-12T08:42:14.936Z
6458970cab9a44f42f620a80
/avatars/f9779b0621c931f922440fec95342444.svg
chaojie mao
user
chaojiemao
67d04a5d7c367a40ae3e417d
Jingfeng Zhang
67d04a5d7c367a40ae3e417e
Yulin Pan
67d04a5d7c367a40ae3e417f
Yu Liu
VACE: All-in-One Video Creation and Editing
Diffusion Transformer has demonstrated powerful capability and scalability in
generating high-quality images and videos. Further pursuing the unification of
generation and editing tasks has yielded significant progress in the domain of
image content creation. However, due to the intrinsic demands for consistency
across both temporal and spatial dynamics, achieving a unified approach for
video synthesis remains challenging. We introduce VACE, which enables users to
perform Video tasks within an All-in-one framework for Creation and Editing.
These tasks include reference-to-video generation, video-to-video editing, and
masked video-to-video editing. Specifically, we effectively integrate the
requirements of various tasks by organizing video task inputs, such as editing,
reference, and masking, into a unified interface referred to as the Video
Condition Unit (VCU). Furthermore, by utilizing a Context Adapter structure, we
inject different task concepts into the model using formalized representations
of temporal and spatial dimensions, allowing it to handle arbitrary video
synthesis tasks flexibly. Extensive experiments demonstrate that the unified
model of VACE achieves performance on par with task-specific models across
various subtasks. Simultaneously, it enables diverse applications through
versatile task combinations. Project page:
https://ali-vilab.github.io/VACE-Page/.
VACE, an all-in-one framework for video creation and editing, integrates multiple tasks within a unified model using a Video Condition Unit and Context Adapter for flexible and consistent video synthesis.
diffusion transformer video synthesis reference-to-video generation video-to-video editing masked video-to-video editing video condition unit context adapter