MotionMaster: Training-free Camera Motion Transfer For Video Generation
The emergence of diffusion models has greatly propelled the progress in image
and video generation. Recently, some efforts have been made in controllable
video generation, including text-to-video generation and video motion control,
among which camera motion control is an important topic. However, existing
camera motion control methods rely on training a temporal camera module, and
necessitate substantial computation resources due to the large amount of
parameters in video generation models. Moreover, existing methods pre-define
camera motion types during training, which limits their flexibility in camera
control. Therefore, to reduce training costs and achieve flexible camera
control, we propose COMD, a novel training-free video motion transfer model,
which disentangles camera motions and object motions in source videos and
transfers the extracted camera motions to new videos. We first propose a
one-shot camera motion disentanglement method to extract camera motion from a
single source video, which separates the moving objects from the background and
estimates the camera motion in the moving objects region based on the motion in
the background by solving a Poisson equation. Furthermore, we propose a
few-shot camera motion disentanglement method to extract the common camera
motion from multiple videos with similar camera motions, which employs a
window-based clustering technique to extract the common features in temporal
attention maps of multiple videos. Finally, we propose a motion combination
method to combine different types of camera motions together, enabling our
model a more controllable and flexible camera control. Extensive experiments
demonstrate that our training-free approach can effectively decouple
camera-object motion and apply the decoupled camera motion to a wide range of
controllable video generation tasks, achieving flexible and diverse camera
motion control.
662a76a2bd72ddcd024848d1
Teng Hu
662a76a2bd72ddcd024848d2
Jiangning Zhang
662a76a2bd72ddcd024848d3
Ran Yi
662a76a2bd72ddcd024848d4
Yating Wang
662a76a2bd72ddcd024848d5
Hongrui Huang
662a76a2bd72ddcd024848d6
Jieyu Weng
662a76a2bd72ddcd024848d7
Yabiao Wang
662a76a2bd72ddcd024848d8
Lizhuang Ma
MotionMaster: Training-free Camera Motion Transfer For Video Generation
The emergence of diffusion models has greatly propelled the progress in image
and video generation. Recently, some efforts have been made in controllable
video generation, including text-to-video generation and video motion control,
among which camera motion control is an important topic. However, existing
camera motion control methods rely on training a temporal camera module, and
necessitate substantial computation resources due to the large amount of
parameters in video generation models. Moreover, existing methods pre-define
camera motion types during training, which limits their flexibility in camera
control. Therefore, to reduce training costs and achieve flexible camera
control, we propose COMD, a novel training-free video motion transfer model,
which disentangles camera motions and object motions in source videos and
transfers the extracted camera motions to new videos. We first propose a
one-shot camera motion disentanglement method to extract camera motion from a
single source video, which separates the moving objects from the background and
estimates the camera motion in the moving objects region based on the motion in
the background by solving a Poisson equation. Furthermore, we propose a
few-shot camera motion disentanglement method to extract the common camera
motion from multiple videos with similar camera motions, which employs a
window-based clustering technique to extract the common features in temporal
attention maps of multiple videos. Finally, we propose a motion combination
method to combine different types of camera motions together, enabling our
model a more controllable and flexible camera control. Extensive experiments
demonstrate that our training-free approach can effectively decouple
camera-object motion and apply the decoupled camera motion to a wide range of
controllable video generation tasks, achieving flexible and diverse camera
motion control.
A novel training-free video motion transfer model, COMD, decouples camera and object motions for flexible and efficient video generation.
diffusion models video generation text-to-video generation video motion control camera motion control temporal camera module Poisson equation window-based clustering temporal attention maps motion combination camera-object motion disentanglement