Shape-for-Motion: Precise and Consistent Video Editing with 3D Proxy
Recent advances in deep generative modeling have unlocked unprecedented
opportunities for video synthesis. In real-world applications, however, users
often seek tools to faithfully realize their creative editing intentions with
precise and consistent control. Despite the progress achieved by existing
methods, ensuring fine-grained alignment with user intentions remains an open
and challenging problem. In this work, we present Shape-for-Motion, a novel
framework that incorporates a 3D proxy for precise and consistent video
editing. Shape-for-Motion achieves this by converting the target object in the
input video to a time-consistent mesh, i.e., a 3D proxy, allowing edits to be
performed directly on the proxy and then inferred back to the video frames. To
simplify the editing process, we design a novel Dual-Propagation Strategy that
allows users to perform edits on the 3D mesh of a single frame, and the edits
are then automatically propagated to the 3D meshes of the other frames. The 3D
meshes for different frames are further projected onto the 2D space to produce
the edited geometry and texture renderings, which serve as inputs to a
decoupled video diffusion model for generating edited results. Our framework
supports various precise and physically-consistent manipulations across the
video frames, including pose editing, rotation, scaling, translation, texture
modification, and object composition. Our approach marks a key step toward
high-quality, controllable video editing workflows. Extensive experiments
demonstrate the superiority and effectiveness of our approach. Project page:
https://shapeformotion.github.io/
6861e87a9e7509383d29ab1e
Yuhao Liu
claimed_verified
2025-06-30T14:03:52.940Z
6351463b8445bbe32e944f6c
/avatars/ec0e8f378d5314d4af97d6c488771b3d.svg
Yuhao Liu
user
LeoLau
6861e87a9e7509383d29ab1f
Tengfei Wang
6861e87a9e7509383d29ab20
Fang Liu
6861e87a9e7509383d29ab21
Zhenwei Wang
claimed_verified
2025-07-31T08:16:16.071Z
6305e9471e731505a76b9099
https://cdn-avatars.huggingface.co/v1/production/uploads/6305e9471e731505a76b9099/w9csjXIEUzrd0sa1FxIXm.jpeg
Zhenwei WANG (Zavier)
user
ZhenweiWang
6861e87a9e7509383d29ab22
Rynson W. H. Lau
Shape-for-Motion: Precise and Consistent Video Editing with 3D Proxy
Recent advances in deep generative modeling have unlocked unprecedented
opportunities for video synthesis. In real-world applications, however, users
often seek tools to faithfully realize their creative editing intentions with
precise and consistent control. Despite the progress achieved by existing
methods, ensuring fine-grained alignment with user intentions remains an open
and challenging problem. In this work, we present Shape-for-Motion, a novel
framework that incorporates a 3D proxy for precise and consistent video
editing. Shape-for-Motion achieves this by converting the target object in the
input video to a time-consistent mesh, i.e., a 3D proxy, allowing edits to be
performed directly on the proxy and then inferred back to the video frames. To
simplify the editing process, we design a novel Dual-Propagation Strategy that
allows users to perform edits on the 3D mesh of a single frame, and the edits
are then automatically propagated to the 3D meshes of the other frames. The 3D
meshes for different frames are further projected onto the 2D space to produce
the edited geometry and texture renderings, which serve as inputs to a
decoupled video diffusion model for generating edited results. Our framework
supports various precise and physically-consistent manipulations across the
video frames, including pose editing, rotation, scaling, translation, texture
modification, and object composition. Our approach marks a key step toward
high-quality, controllable video editing workflows. Extensive experiments
demonstrate the superiority and effectiveness of our approach. Project page:
https://shapeformotion.github.io/
A novel framework integrates 3D proxy meshes and a decoupled video diffusion model to achieve precise and consistent video editing.
3D proxy Dual-Propagation Strategy 3D meshes 2D space video diffusion model pose editing rotation scaling translation texture modification object composition