title: MotionCtrl: A Unified and Flexible Motion Controller for Video
    Generation
summary: Motions in a video primarily consist of camera motion, induced by camera
  movement, and object motion, resulting from object movement. Accurate control
  of both camera and object motion is essential for video generation. However,
  existing works either mainly focus on one type of motion or do not clearly
  distinguish between the two, limiting their control capabilities and diversity.
  Therefore, this paper presents MotionCtrl, a unified and flexible motion
  controller for video generation designed to effectively and independently
  control camera and object motion. The architecture and training strategy of
  MotionCtrl are carefully devised, taking into account the inherent properties
  of camera motion, object motion, and imperfect training data. Compared to
  previous methods, MotionCtrl offers three main advantages: 1) It effectively
  and independently controls camera motion and object motion, enabling more
  fine-grained motion control and facilitating flexible and diverse combinations
  of both types of motion. 2) Its motion conditions are determined by camera
  poses and trajectories, which are appearance-free and minimally impact the
  appearance or shape of objects in generated videos. 3) It is a relatively
  generalizable model that can adapt to a wide array of camera poses and
  trajectories once trained. Extensive qualitative and quantitative experiments
  have been conducted to demonstrate the superiority of MotionCtrl over existing
  methods.
paper_authors._id: 657136ad87a92b76924dc031
paper_authors.name: Zhouxia Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-07T11:45:22.824Z
paper_authors._id: 64db92a5858f8a41c11669b7
paper_authors.avatarUrl: /avatars/e834d8f1d4781e3bb0b5d6d25b3b3505.svg
paper_authors.fullname: Zhouxia Wang
paper_authors.type: user
paper_authors.user: wzhouxiff
paper_authors._id: 657136ad87a92b76924dc032
paper_authors.name: Ziyang Yuan
paper_authors._id: 657136ad87a92b76924dc033
paper_authors.name: Xintao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-07T11:45:37.609Z
paper_authors._id: 60e272ca6c78a8c122b12127
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
paper_authors.fullname: Xintao Wang
paper_authors.type: user
paper_authors.user: Xintao
paper_authors._id: 657136ad87a92b76924dc034
paper_authors.name: Tianshui Chen
paper_authors._id: 657136ad87a92b76924dc035
paper_authors.name: Menghan Xia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-07T11:45:57.019Z
paper_authors._id: 63401c89f81b9d101361f712
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665146415483-63401c89f81b9d101361f712.png
paper_authors.fullname: Richard
paper_authors.type: user
paper_authors.user: menghanxia
paper_authors._id: 657136ad87a92b76924dc036
paper_authors.name: Ping Luo
paper_authors._id: 657136ad87a92b76924dc037
paper_authors.name: Ying Shan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-08T08:00:31.063Z
paper_authors._id: 63ca3ddc04c979828310bfcb
paper_authors.avatarUrl: /avatars/615e0d8622950b4408b40d550f02a894.svg
paper_authors.fullname: Ying Shan
paper_authors.type: user
paper_authors.user: yshan2u
paper_title: MotionCtrl: A Unified and Flexible Motion Controller for Video
    Generation
paper_summary: Motions in a video primarily consist of camera motion, induced by camera
  movement, and object motion, resulting from object movement. Accurate control
  of both camera and object motion is essential for video generation. However,
  existing works either mainly focus on one type of motion or do not clearly
  distinguish between the two, limiting their control capabilities and diversity.
  Therefore, this paper presents MotionCtrl, a unified and flexible motion
  controller for video generation designed to effectively and independently
  control camera and object motion. The architecture and training strategy of
  MotionCtrl are carefully devised, taking into account the inherent properties
  of camera motion, object motion, and imperfect training data. Compared to
  previous methods, MotionCtrl offers three main advantages: 1) It effectively
  and independently controls camera motion and object motion, enabling more
  fine-grained motion control and facilitating flexible and diverse combinations
  of both types of motion. 2) Its motion conditions are determined by camera
  poses and trajectories, which are appearance-free and minimally impact the
  appearance or shape of objects in generated videos. 3) It is a relatively
  generalizable model that can adapt to a wide array of camera poses and
  trajectories once trained. Extensive qualitative and quantitative experiments
  have been conducted to demonstrate the superiority of MotionCtrl over existing
  methods.
paper_ai_summary: MotionCtrl is a unified motion controller for video generation that independently controls camera and object motion, offering more flexible and diverse video outputs.
paper_ai_keywords: MotionCtrl camera motion object motion motion conditions camera poses trajectories motion control