title: Motion Prompting: Controlling Video Generation with Motion Trajectories
summary: Motion control is crucial for generating expressive and compelling video
  content; however, most existing video generation models rely mainly on text
  prompts for control, which struggle to capture the nuances of dynamic actions
  and temporal compositions. To this end, we train a video generation model
  conditioned on spatio-temporally sparse or dense motion trajectories. In
  contrast to prior motion conditioning work, this flexible representation can
  encode any number of trajectories, object-specific or global scene motion, and
  temporally sparse motion; due to its flexibility we refer to this conditioning
  as motion prompts. While users may directly specify sparse trajectories, we
  also show how to translate high-level user requests into detailed, semi-dense
  motion prompts, a process we term motion prompt expansion. We demonstrate the
  versatility of our approach through various applications, including camera and
  object motion control, "interacting" with an image, motion transfer, and image
  editing. Our results showcase emergent behaviors, such as realistic physics,
  suggesting the potential of motion prompts for probing video models and
  interacting with future generative world models. Finally, we evaluate
  quantitatively, conduct a human study, and demonstrate strong performance.
  Video results are available on our webpage: https://motion-prompting.github.io/
paper_authors._id: 6750808b94f48729d536d9d1
paper_authors.name: Daniel Geng
paper_authors._id: 6750808b94f48729d536d9d2
paper_authors.name: Charles Herrmann
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T13:58:58.267Z
paper_authors._id: 64c7f088c864d962eddeaf8c
paper_authors.avatarUrl: /avatars/e6364c56e6cb6907b6dd2d34eed83720.svg
paper_authors.fullname: Charles Herrmann
paper_authors.type: user
paper_authors.user: irwinherrmann
paper_authors._id: 6750808b94f48729d536d9d3
paper_authors.name: Junhwa Hur
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T13:58:50.219Z
paper_authors._id: 647f3046b0e96764589e107a
paper_authors.avatarUrl: /avatars/1074ee5ec1b2d5d297406f9d455213c4.svg
paper_authors.fullname: Junhwa Hur
paper_authors.type: user
paper_authors.user: hurjunhwa
paper_authors._id: 6750808b94f48729d536d9d4
paper_authors.name: Forrester Cole
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T13:58:42.494Z
paper_authors._id: 635003011d81beb8e244e68d
paper_authors.avatarUrl: /avatars/32f46e6778d1c52ebdd6f2511ba48257.svg
paper_authors.fullname: Forrester Cole
paper_authors.type: user
paper_authors.user: fcole
paper_authors._id: 6750808b94f48729d536d9d5
paper_authors.name: Serena Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T13:58:36.556Z
paper_authors._id: 64b817e4cf14c2fabe85ef68
paper_authors.avatarUrl: /avatars/88093121b07adbc14c53c504c67709ea.svg
paper_authors.fullname: Serena Zhang
paper_authors.type: user
paper_authors.user: serenaz
paper_authors._id: 6750808b94f48729d536d9d6
paper_authors.name: Tobias Pfaff
paper_authors._id: 6750808b94f48729d536d9d7
paper_authors.name: Tatiana Lopez-Guevara
paper_authors._id: 6750808b94f48729d536d9d8
paper_authors.name: Carl Doersch
paper_authors._id: 6750808b94f48729d536d9d9
paper_authors.name: Yusuf Aytar
paper_authors._id: 6750808b94f48729d536d9da
paper_authors.name: Michael Rubinstein
paper_authors._id: 6750808b94f48729d536d9db
paper_authors.name: Chen Sun
paper_authors._id: 6750808b94f48729d536d9dc
paper_authors.name: Oliver Wang
paper_authors._id: 6750808b94f48729d536d9dd
paper_authors.name: Andrew Owens
paper_authors._id: 6750808b94f48729d536d9de
paper_authors.name: Deqing Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T13:58:06.713Z
paper_authors._id: 6750994d0438fdc8b9f86303
paper_authors.avatarUrl: /avatars/fe89e72a6cd15649df2c7845033fa8ae.svg
paper_authors.fullname: Deqing Sun
paper_authors.type: user
paper_authors.user: deqingsun
paper_title: Motion Prompting: Controlling Video Generation with Motion Trajectories
paper_summary: Motion control is crucial for generating expressive and compelling video
  content; however, most existing video generation models rely mainly on text
  prompts for control, which struggle to capture the nuances of dynamic actions
  and temporal compositions. To this end, we train a video generation model
  conditioned on spatio-temporally sparse or dense motion trajectories. In
  contrast to prior motion conditioning work, this flexible representation can
  encode any number of trajectories, object-specific or global scene motion, and
  temporally sparse motion; due to its flexibility we refer to this conditioning
  as motion prompts. While users may directly specify sparse trajectories, we
  also show how to translate high-level user requests into detailed, semi-dense
  motion prompts, a process we term motion prompt expansion. We demonstrate the
  versatility of our approach through various applications, including camera and
  object motion control, "interacting" with an image, motion transfer, and image
  editing. Our results showcase emergent behaviors, such as realistic physics,
  suggesting the potential of motion prompts for probing video models and
  interacting with future generative world models. Finally, we evaluate
  quantitatively, conduct a human study, and demonstrate strong performance.
  Video results are available on our webpage: https://motion-prompting.github.io/
paper_ai_keywords: spatio-temporally sparse or dense trajectories motion prompts motion prompt expansion camera and object motion control motion transfer image editing emergent behaviors realistic physics video models generative world models