LeviTor: 3D Trajectory Oriented Image-to-Video Synthesis
The intuitive nature of drag-based interaction has led to its growing
adoption for controlling object trajectories in image-to-video synthesis.
Still, existing methods that perform dragging in the 2D space usually face
ambiguity when handling out-of-plane movements. In this work, we augment the
interaction with a new dimension, i.e., the depth dimension, such that users
are allowed to assign a relative depth for each point on the trajectory. That
way, our new interaction paradigm not only inherits the convenience from 2D
dragging, but facilitates trajectory control in the 3D space, broadening the
scope of creativity. We propose a pioneering method for 3D trajectory control
in image-to-video synthesis by abstracting object masks into a few cluster
points. These points, accompanied by the depth information and the instance
information, are finally fed into a video diffusion model as the control
signal. Extensive experiments validate the effectiveness of our approach,
dubbed LeviTor, in precisely manipulating the object movements when producing
photo-realistic videos from static images. Project page:
https://ppetrichor.github.io/levitor.github.io/
6764f548cee1fdbd9765e9bc
Hanlin Wang
claimed_verified
2024-12-20T08:32:36.964Z
665f059a8947302aa2c63afe
/avatars/50f560285946532321a0bd526494148d.svg
hanlin wang
user
hlwang06
6764f548cee1fdbd9765e9bd
Hao Ouyang
6764f548cee1fdbd9765e9be
Qiuyu Wang
claimed_verified
2024-12-20T08:32:28.937Z
64981bea09cea550852652af
/avatars/df528e9008972c8e5ae4d278e617476c.svg
Qiuyu Wang
user
qiuyuu
6764f548cee1fdbd9765e9bf
Wen Wang
6764f548cee1fdbd9765e9c0
Ka Leong Cheng
admin_assigned
2024-12-20T09:26:43.630Z
64acd2ec39fcfebff8c79c00
/avatars/9419384846b92182f2c47ce2fbd0f8d3.svg
Ka Leong Cheng
user
felixcheng97
6764f548cee1fdbd9765e9c1
Qifeng Chen
admin_assigned
2024-12-20T09:26:37.950Z
6467b121e7a6a374fd19b44b
/avatars/3f2874d58986d651aef55e3408b05700.svg
Qifeng Chen
user
cqf
6764f548cee1fdbd9765e9c2
Yujun Shen
6764f548cee1fdbd9765e9c3
Limin Wang
admin_assigned
2024-12-20T09:26:23.360Z
62c77f4352d8ae531f5511f9
/avatars/50198ccb02ccd286975a4613fbabee28.svg
Limin Wang
user
lmwang
LeviTor: 3D Trajectory Oriented Image-to-Video Synthesis
The intuitive nature of drag-based interaction has led to its growing
adoption for controlling object trajectories in image-to-video synthesis.
Still, existing methods that perform dragging in the 2D space usually face
ambiguity when handling out-of-plane movements. In this work, we augment the
interaction with a new dimension, i.e., the depth dimension, such that users
are allowed to assign a relative depth for each point on the trajectory. That
way, our new interaction paradigm not only inherits the convenience from 2D
dragging, but facilitates trajectory control in the 3D space, broadening the
scope of creativity. We propose a pioneering method for 3D trajectory control
in image-to-video synthesis by abstracting object masks into a few cluster
points. These points, accompanied by the depth information and the instance
information, are finally fed into a video diffusion model as the control
signal. Extensive experiments validate the effectiveness of our approach,
dubbed LeviTor, in precisely manipulating the object movements when producing
photo-realistic videos from static images. Project page:
https://ppetrichor.github.io/levitor.github.io/
A 3D drag-based interaction method that uses object masks and depth information to control trajectories in image-to-video synthesis, enhancing creativity and precision.
drag-based interaction 3D trajectory control image-to-video synthesis object masks cluster points depth information instance information video diffusion model LeviTor