  Motions in a video primarily consist of camera motion, induced by camera
movement, and object motion, resulting from object movement. Accurate control
of both camera and object motion is essential for video generation. However,
existing works either mainly focus on one type of motion or do not clearly
distinguish between the two, limiting their control capabilities and diversity.
Therefore, this paper presents MotionCtrl, a unified and flexible motion
controller for video generation designed to effectively and independently
control camera and object motion. The architecture and training strategy of
MotionCtrl are carefully devised, taking into account the inherent properties
of camera motion, object motion, and imperfect training data. Compared to
previous methods, MotionCtrl offers three main advantages: 1) It effectively
and independently controls camera motion and object motion, enabling more
fine-grained motion control and facilitating flexible and diverse combinations
of both types of motion. 2) Its motion conditions are determined by camera
poses and trajectories, which are appearance-free and minimally impact the
appearance or shape of objects in generated videos. 3) It is a relatively
generalizable model that can adapt to a wide array of camera poses and
trajectories once trained. Extensive qualitative and quantitative experiments
have been conducted to demonstrate the superiority of MotionCtrl over existing
methods.

Computer Vision and Pattern Recognition;Artificial Intelligence;Machine Learning;Multimedia