title: MotionMaster: Training-free Camera Motion Transfer For Video Generation
summary: The emergence of diffusion models has greatly propelled the progress in image
  and video generation. Recently, some efforts have been made in controllable
  video generation, including text-to-video generation and video motion control,
  among which camera motion control is an important topic. However, existing
  camera motion control methods rely on training a temporal camera module, and
  necessitate substantial computation resources due to the large amount of
  parameters in video generation models. Moreover, existing methods pre-define
  camera motion types during training, which limits their flexibility in camera
  control. Therefore, to reduce training costs and achieve flexible camera
  control, we propose COMD, a novel training-free video motion transfer model,
  which disentangles camera motions and object motions in source videos and
  transfers the extracted camera motions to new videos. We first propose a
  one-shot camera motion disentanglement method to extract camera motion from a
  single source video, which separates the moving objects from the background and
  estimates the camera motion in the moving objects region based on the motion in
  the background by solving a Poisson equation. Furthermore, we propose a
  few-shot camera motion disentanglement method to extract the common camera
  motion from multiple videos with similar camera motions, which employs a
  window-based clustering technique to extract the common features in temporal
  attention maps of multiple videos. Finally, we propose a motion combination
  method to combine different types of camera motions together, enabling our
  model a more controllable and flexible camera control. Extensive experiments
  demonstrate that our training-free approach can effectively decouple
  camera-object motion and apply the decoupled camera motion to a wide range of
  controllable video generation tasks, achieving flexible and diverse camera
  motion control.
paper_authors._id: 662a76a2bd72ddcd024848d1
paper_authors.name: Teng Hu
paper_authors._id: 662a76a2bd72ddcd024848d2
paper_authors.name: Jiangning Zhang
paper_authors._id: 662a76a2bd72ddcd024848d3
paper_authors.name: Ran Yi
paper_authors._id: 662a76a2bd72ddcd024848d4
paper_authors.name: Yating Wang
paper_authors._id: 662a76a2bd72ddcd024848d5
paper_authors.name: Hongrui Huang
paper_authors._id: 662a76a2bd72ddcd024848d6
paper_authors.name: Jieyu Weng
paper_authors._id: 662a76a2bd72ddcd024848d7
paper_authors.name: Yabiao Wang
paper_authors._id: 662a76a2bd72ddcd024848d8
paper_authors.name: Lizhuang Ma
paper_title: MotionMaster: Training-free Camera Motion Transfer For Video Generation
paper_summary: The emergence of diffusion models has greatly propelled the progress in image
  and video generation. Recently, some efforts have been made in controllable
  video generation, including text-to-video generation and video motion control,
  among which camera motion control is an important topic. However, existing
  camera motion control methods rely on training a temporal camera module, and
  necessitate substantial computation resources due to the large amount of
  parameters in video generation models. Moreover, existing methods pre-define
  camera motion types during training, which limits their flexibility in camera
  control. Therefore, to reduce training costs and achieve flexible camera
  control, we propose COMD, a novel training-free video motion transfer model,
  which disentangles camera motions and object motions in source videos and
  transfers the extracted camera motions to new videos. We first propose a
  one-shot camera motion disentanglement method to extract camera motion from a
  single source video, which separates the moving objects from the background and
  estimates the camera motion in the moving objects region based on the motion in
  the background by solving a Poisson equation. Furthermore, we propose a
  few-shot camera motion disentanglement method to extract the common camera
  motion from multiple videos with similar camera motions, which employs a
  window-based clustering technique to extract the common features in temporal
  attention maps of multiple videos. Finally, we propose a motion combination
  method to combine different types of camera motions together, enabling our
  model a more controllable and flexible camera control. Extensive experiments
  demonstrate that our training-free approach can effectively decouple
  camera-object motion and apply the decoupled camera motion to a wide range of
  controllable video generation tasks, achieving flexible and diverse camera
  motion control.
paper_ai_keywords: diffusion models alpha matte image generation video generation text-to-video generation video motion control camera motion control pre-trained camera module temporal dependency modeling video generation models training-free video motion transfer model camera motion disentanglement object motion disentanglement Poisson equation few-shot camera motion disentanglement window-based clustering technique temporal attention maps camera motions combination camera-object motion decoupling controllable video generation tasks