title: Motion Control for Enhanced Complex Action Video Generation
summary: Existing text-to-video (T2V) models often struggle with generating videos
  with sufficiently pronounced or complex actions. A key limitation lies in the
  text prompt's inability to precisely convey intricate motion details. To
  address this, we propose a novel framework, MVideo, designed to produce
  long-duration videos with precise, fluid actions. MVideo overcomes the
  limitations of text prompts by incorporating mask sequences as an additional
  motion condition input, providing a clearer, more accurate representation of
  intended actions. Leveraging foundational vision models such as GroundingDINO
  and SAM2, MVideo automatically generates mask sequences, enhancing both
  efficiency and robustness. Our results demonstrate that, after training, MVideo
  effectively aligns text prompts with motion conditions to produce videos that
  simultaneously meet both criteria. This dual control mechanism allows for more
  dynamic video generation by enabling alterations to either the text prompt or
  motion condition independently, or both in tandem. Furthermore, MVideo supports
  motion condition editing and composition, facilitating the generation of videos
  with more complex actions. MVideo thus advances T2V motion generation, setting
  a strong benchmark for improved action depiction in current video diffusion
  models. Our project page is available at https://mvideo-v1.github.io/.
paper_authors._id: 6736121f12b0cbbb1232b84f
paper_authors.name: Qiang Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T14:33:25.630Z
paper_authors._id: 64db1edc7266618e853c6c13
paper_authors.avatarUrl: /avatars/b3362a882fd9cee3d9f59d8dbcc02367.svg
paper_authors.fullname: mightyzau
paper_authors.type: user
paper_authors.user: mightyzau
paper_authors._id: 6736121f12b0cbbb1232b850
paper_authors.name: Shaofeng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-15T14:24:41.026Z
paper_authors._id: 669912b8a22692f3544c8ec2
paper_authors.avatarUrl: /avatars/3e4f90b4bf9c0e6e8dcf0bc8bd9d8cfa.svg
paper_authors.fullname: Shaofeng Zhang
paper_authors.type: user
paper_authors.user: SimonZ1881
paper_authors._id: 6736121f12b0cbbb1232b851
paper_authors.name: Nianzu Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-15T14:24:47.634Z
paper_authors._id: 66371a41164dfca129939a72
paper_authors.avatarUrl: /avatars/2fdf0a059e06643f9f92c57efd1610b5.svg
paper_authors.fullname: Nianzu Yang
paper_authors.type: user
paper_authors.user: yangnianzu
paper_authors._id: 6736121f12b0cbbb1232b852
paper_authors.name: Ye Qian
paper_authors._id: 6736121f12b0cbbb1232b853
paper_authors.name: Hao Li
paper_title: Motion Control for Enhanced Complex Action Video Generation
paper_summary: Existing text-to-video (T2V) models often struggle with generating videos
  with sufficiently pronounced or complex actions. A key limitation lies in the
  text prompt's inability to precisely convey intricate motion details. To
  address this, we propose a novel framework, MVideo, designed to produce
  long-duration videos with precise, fluid actions. MVideo overcomes the
  limitations of text prompts by incorporating mask sequences as an additional
  motion condition input, providing a clearer, more accurate representation of
  intended actions. Leveraging foundational vision models such as GroundingDINO
  and SAM2, MVideo automatically generates mask sequences, enhancing both
  efficiency and robustness. Our results demonstrate that, after training, MVideo
  effectively aligns text prompts with motion conditions to produce videos that
  simultaneously meet both criteria. This dual control mechanism allows for more
  dynamic video generation by enabling alterations to either the text prompt or
  motion condition independently, or both in tandem. Furthermore, MVideo supports
  motion condition editing and composition, facilitating the generation of videos
  with more complex actions. MVideo thus advances T2V motion generation, setting
  a strong benchmark for improved action depiction in current video diffusion
  models. Our project page is available at https://mvideo-v1.github.io/.
paper_ai_keywords: text-to-video (T2V) models mask sequences motion condition input GroundingDINO SAM2 dual control mechanism motion condition editing composition video diffusion models