Enabling Versatile Controls for Video Diffusion Models
Despite substantial progress in text-to-video generation, achieving precise
and flexible control over fine-grained spatiotemporal attributes remains a
significant unresolved challenge in video generation research. To address these
limitations, we introduce VCtrl (also termed PP-VCtrl), a novel framework
designed to enable fine-grained control over pre-trained video diffusion models
in a unified manner. VCtrl integrates diverse user-specified control
signals-such as Canny edges, segmentation masks, and human keypoints-into
pretrained video diffusion models via a generalizable conditional module
capable of uniformly encoding multiple types of auxiliary signals without
modifying the underlying generator. Additionally, we design a unified control
signal encoding pipeline and a sparse residual connection mechanism to
efficiently incorporate control representations. Comprehensive experiments and
human evaluations demonstrate that VCtrl effectively enhances controllability
and generation quality. The source code and pre-trained models are publicly
available and implemented using the PaddlePaddle framework at
http://github.com/PaddlePaddle/PaddleMIX/tree/develop/ppdiffusers/examples/ppvctrl.
67e0c303ff27a08e3896134a
Xu Zhang
67e0c303ff27a08e3896134b
Hao Zhou
67e0c303ff27a08e3896134c
Haoming Qin
67e0c303ff27a08e3896134d
Xiaobin Lu
admin_assigned
2025-03-24T12:47:23.819Z
64a42dc94c10e3675498ad92
/avatars/1f3e027d27af8444a437399d9ee6dd8f.svg
xiaobin Lu
user
xilu5047
67e0c303ff27a08e3896134e
Jiaxing Yan
admin_assigned
2025-03-24T12:46:19.494Z
67c4610fa8ec9d71bf3b2a48
/avatars/88eb9641e1dc7a237498c5257a0232aa.svg
yanjiaxing
user
YanJiaXing
67e0c303ff27a08e3896134f
Guanzhong Wang
admin_assigned
2025-03-24T12:47:37.219Z
677f5eca19fb7f8c452e0876
/avatars/f053709ece377164975e266e52ab23af.svg
wang guanzhong
user
jerrywgz
67e0c303ff27a08e38961350
Zeyu Chen
67e0c303ff27a08e38961351
Yi Liu
Enabling Versatile Controls for Video Diffusion Models
Despite substantial progress in text-to-video generation, achieving precise
and flexible control over fine-grained spatiotemporal attributes remains a
significant unresolved challenge in video generation research. To address these
limitations, we introduce VCtrl (also termed PP-VCtrl), a novel framework
designed to enable fine-grained control over pre-trained video diffusion models
in a unified manner. VCtrl integrates diverse user-specified control
signals-such as Canny edges, segmentation masks, and human keypoints-into
pretrained video diffusion models via a generalizable conditional module
capable of uniformly encoding multiple types of auxiliary signals without
modifying the underlying generator. Additionally, we design a unified control
signal encoding pipeline and a sparse residual connection mechanism to
efficiently incorporate control representations. Comprehensive experiments and
human evaluations demonstrate that VCtrl effectively enhances controllability
and generation quality. The source code and pre-trained models are publicly
available and implemented using the PaddlePaddle framework at
http://github.com/PaddlePaddle/PaddleMIX/tree/develop/ppdiffusers/examples/ppvctrl.
VCtrl enhances control over fine-grained spatiotemporal attributes in text-to-video generation by integrating various control signals into pre-trained video diffusion models without modifying the generator.
text-to-video generation fine-grained spatiotemporal attributes VCtrl PP-VCtrl Canny edges segmentation masks human keypoints conditional module auxiliary signals unified control signal encoding pipeline sparse residual connection mechanism camera diffusion models generation quality