ObjCtrl-2.5D: Training-free Object Control with Camera Poses
This study aims to achieve more precise and versatile object control in
image-to-video (I2V) generation. Current methods typically represent the
spatial movement of target objects with 2D trajectories, which often fail to
capture user intention and frequently produce unnatural results. To enhance
control, we present ObjCtrl-2.5D, a training-free object control approach that
uses a 3D trajectory, extended from a 2D trajectory with depth information, as
a control signal. By modeling object movement as camera movement, ObjCtrl-2.5D
represents the 3D trajectory as a sequence of camera poses, enabling object
motion control using an existing camera motion control I2V generation model
(CMC-I2V) without training. To adapt the CMC-I2V model originally designed for
global motion control to handle local object motion, we introduce a module to
isolate the target object from the background, enabling independent local
control. In addition, we devise an effective way to achieve more accurate
object control by sharing low-frequency warped latent within the object's
region across frames. Extensive experiments demonstrate that ObjCtrl-2.5D
significantly improves object control accuracy compared to training-free
methods and offers more diverse control capabilities than training-based
approaches using 2D trajectories, enabling complex effects like object
rotation. Code and results are available at
https://wzhouxiff.github.io/projects/ObjCtrl-2.5D/.
6759180441b35c9329025a5e
Zhouxia Wang
admin_assigned
2024-12-11T13:50:16.745Z
64db92a5858f8a41c11669b7
/avatars/e834d8f1d4781e3bb0b5d6d25b3b3505.svg
Zhouxia Wang
user
wzhouxiff
6759180441b35c9329025a5f
Yushi Lan
admin_assigned
2024-12-11T13:50:10.996Z
64b55f62e6b0c53f97e64034
/avatars/d5fc53de37d4fb0b5095a5fb21b43503.svg
LAN YUSHI
user
yslan
6759180441b35c9329025a60
Shangchen Zhou
admin_assigned
2024-12-11T13:50:00.183Z
62e57662ae9d3f10acbb1b1b
https://cdn-avatars.huggingface.co/v1/production/uploads/62e57662ae9d3f10acbb1b1b/lg58jdbNyv6LGH2LFnZDF.png
Shangchen Zhou
user
sczhou
6759180441b35c9329025a61
Chen Change Loy
admin_assigned
2024-12-11T13:49:54.135Z
67459d997a49660f7f62452f
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/X25yTdxC_hcLVxpx4voZo.png
Chen Change Loy
user
cavanloy
ObjCtrl-2.5D: Training-free Object Control with Camera Poses
This study aims to achieve more precise and versatile object control in
image-to-video (I2V) generation. Current methods typically represent the
spatial movement of target objects with 2D trajectories, which often fail to
capture user intention and frequently produce unnatural results. To enhance
control, we present ObjCtrl-2.5D, a training-free object control approach that
uses a 3D trajectory, extended from a 2D trajectory with depth information, as
a control signal. By modeling object movement as camera movement, ObjCtrl-2.5D
represents the 3D trajectory as a sequence of camera poses, enabling object
motion control using an existing camera motion control I2V generation model
(CMC-I2V) without training. To adapt the CMC-I2V model originally designed for
global motion control to handle local object motion, we introduce a module to
isolate the target object from the background, enabling independent local
control. In addition, we devise an effective way to achieve more accurate
object control by sharing low-frequency warped latent within the object's
region across frames. Extensive experiments demonstrate that ObjCtrl-2.5D
significantly improves object control accuracy compared to training-free
methods and offers more diverse control capabilities than training-based
approaches using 2D trajectories, enabling complex effects like object
rotation. Code and results are available at
https://wzhouxiff.github.io/projects/ObjCtrl-2.5D/.
ObjCtrl-2.5D enhances image-to-video object control using 3D trajectories represented as camera poses, allowing precise and diverse control without training.
image-to-video generation 2D trajectory 3D trajectory depth information camera motion control CMC-I2V local object motion background isolation low-frequency warped latent object control accuracy