Uni3C: Unifying Precisely 3D-Enhanced Camera and Human Motion Controls
  for Video Generation
Camera and human motion controls have been extensively studied for video
generation, but existing approaches typically address them separately,
suffering from limited data with high-quality annotations for both aspects. To
overcome this, we present Uni3C, a unified 3D-enhanced framework for precise
control of both camera and human motion in video generation. Uni3C includes two
key contributions. First, we propose a plug-and-play control module trained
with a frozen video generative backbone, PCDController, which utilizes
unprojected point clouds from monocular depth to achieve accurate camera
control. By leveraging the strong 3D priors of point clouds and the powerful
capacities of video foundational models, PCDController shows impressive
generalization, performing well regardless of whether the inference backbone is
frozen or fine-tuned. This flexibility enables different modules of Uni3C to be
trained in specific domains, i.e., either camera control or human motion
control, reducing the dependency on jointly annotated data. Second, we propose
a jointly aligned 3D world guidance for the inference phase that seamlessly
integrates both scenic point clouds and SMPL-X characters to unify the control
signals for camera and human motion, respectively. Extensive experiments
confirm that PCDController enjoys strong robustness in driving camera motion
for fine-tuned backbones of video generation. Uni3C substantially outperforms
competitors in both camera controllability and human motion quality.
Additionally, we collect tailored validation sets featuring challenging camera
movements and human actions to validate the effectiveness of our method.
68075a2bdadb28dddc13dfc6
Chenjie Cao
admin_assigned
2025-04-22T11:03:42.731Z
64892bc82eafb7b91182bec5
/avatars/c9eed91e60f3ea227b35e111d3fc4200.svg
chenjie cao
user
ewrfcas
68075a2bdadb28dddc13dfc7
Jingkai Zhou
admin_assigned
2025-04-22T11:03:50.199Z
6434caa64b34368fdb07da48
/avatars/8da3d3d1e274ff4ad409234678b1b952.svg
Jingkai Zhou
user
theFoxofSky
68075a2bdadb28dddc13dfc8
Shikai Li
68075a2bdadb28dddc13dfc9
Jingyun Liang
68075a2bdadb28dddc13dfca
Chaohui Yu
68075a2bdadb28dddc13dfcb
Fan Wang
68075a2bdadb28dddc13dfcc
Xiangyang Xue
68075a2bdadb28dddc13dfcd
Yanwei Fu
admin_assigned
2025-04-22T11:04:23.211Z
6409fcc8f3dabf93824c84c6
/avatars/dd8fd579630e50ba3058c3829604478e.svg
YANWEI
user
yanweifuture
Uni3C: Unifying Precisely 3D-Enhanced Camera and Human Motion Controls
  for Video Generation
Camera and human motion controls have been extensively studied for video
generation, but existing approaches typically address them separately,
suffering from limited data with high-quality annotations for both aspects. To
overcome this, we present Uni3C, a unified 3D-enhanced framework for precise
control of both camera and human motion in video generation. Uni3C includes two
key contributions. First, we propose a plug-and-play control module trained
with a frozen video generative backbone, PCDController, which utilizes
unprojected point clouds from monocular depth to achieve accurate camera
control. By leveraging the strong 3D priors of point clouds and the powerful
capacities of video foundational models, PCDController shows impressive
generalization, performing well regardless of whether the inference backbone is
frozen or fine-tuned. This flexibility enables different modules of Uni3C to be
trained in specific domains, i.e., either camera control or human motion
control, reducing the dependency on jointly annotated data. Second, we propose
a jointly aligned 3D world guidance for the inference phase that seamlessly
integrates both scenic point clouds and SMPL-X characters to unify the control
signals for camera and human motion, respectively. Extensive experiments
confirm that PCDController enjoys strong robustness in driving camera motion
for fine-tuned backbones of video generation. Uni3C substantially outperforms
competitors in both camera controllability and human motion quality.
Additionally, we collect tailored validation sets featuring challenging camera
movements and human actions to validate the effectiveness of our method.
Uni3C, a unified 3D-enhanced framework, improves camera and human motion control in video generation by leveraging unprojected point clouds and SMPL-X characters for robust and precise control.
Uni3C PCDController monocular depth point clouds video generative backbone 3D priors SMPL-X 3D world guidance camera control human motion control