title: DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise
    Motion Control
summary: Recent advances in customized video generation have enabled users to create
  videos tailored to both specific subjects and motion trajectories. However,
  existing methods often require complicated test-time fine-tuning and struggle
  with balancing subject learning and motion control, limiting their real-world
  applications. In this paper, we present DreamVideo-2, a zero-shot video
  customization framework capable of generating videos with a specific subject
  and motion trajectory, guided by a single image and a bounding box sequence,
  respectively, and without the need for test-time fine-tuning. Specifically, we
  introduce reference attention, which leverages the model's inherent
  capabilities for subject learning, and devise a mask-guided motion module to
  achieve precise motion control by fully utilizing the robust motion signal of
  box masks derived from bounding boxes. While these two components achieve their
  intended functions, we empirically observe that motion control tends to
  dominate over subject learning. To address this, we propose two key designs: 1)
  the masked reference attention, which integrates a blended latent mask modeling
  scheme into reference attention to enhance subject representations at the
  desired positions, and 2) a reweighted diffusion loss, which differentiates the
  contributions of regions inside and outside the bounding boxes to ensure a
  balance between subject and motion control. Extensive experimental results on a
  newly curated dataset demonstrate that DreamVideo-2 outperforms
  state-of-the-art methods in both subject customization and motion control. The
  dataset, code, and models will be made publicly available.
paper_authors._id: 6711c8f0fe58fd40dde8c60a
paper_authors.name: Yujie Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T12:42:53.127Z
paper_authors._id: 637f70d6fab5db9101c3dfc8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637f70d6fab5db9101c3dfc8/NgkYNXWLDavLbrnCby2Fl.jpeg
paper_authors.fullname: Yujie Wei
paper_authors.type: user
paper_authors.user: weilllllls
paper_authors._id: 6711c8f0fe58fd40dde8c60b
paper_authors.name: Shiwei Zhang
paper_authors._id: 6711c8f0fe58fd40dde8c60c
paper_authors.name: Hangjie Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T12:53:28.821Z
paper_authors._id: 649d54b314afbb10ce2a9eeb
paper_authors.avatarUrl: /avatars/15c325d8c2273ff63569f23015e98486.svg
paper_authors.fullname: Hangjie Yuan
paper_authors.type: user
paper_authors.user: JacobYuan
paper_authors._id: 6711c8f0fe58fd40dde8c60d
paper_authors.name: Xiang Wang
paper_authors._id: 6711c8f0fe58fd40dde8c60e
paper_authors.name: Haonan Qiu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T12:52:59.555Z
paper_authors._id: 63023b6ab002e9a4a2152890
paper_authors.avatarUrl: /avatars/cae8ba0a8d61fb4e576934431f43991b.svg
paper_authors.fullname: Haonan Qiu
paper_authors.type: user
paper_authors.user: MoonQiu
paper_authors._id: 6711c8f0fe58fd40dde8c60f
paper_authors.name: Rui Zhao
paper_authors._id: 6711c8f0fe58fd40dde8c610
paper_authors.name: Yutong Feng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T12:53:05.174Z
paper_authors._id: 64a54e468cfaa458bd6844bf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64a54e468cfaa458bd6844bf/5Gmf4tAr59GNl-2VaZDbu.png
paper_authors.fullname: Yutong Feng
paper_authors.type: user
paper_authors.user: fengyutong
paper_authors._id: 6711c8f0fe58fd40dde8c611
paper_authors.name: Feng Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-02T09:41:00.897Z
paper_authors._id: 661658b9e398e4f0757221ce
paper_authors.avatarUrl: /avatars/9e9d55145661d00e2a9e029ba2e085d8.svg
paper_authors.fullname: Liu Feng
paper_authors.type: user
paper_authors.user: LiewFeng
paper_authors._id: 6711c8f0fe58fd40dde8c612
paper_authors.name: Zhizhong Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T12:53:11.406Z
paper_authors._id: 64c4aad5d68946edad616413
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c4aad5d68946edad616413/RY2lSiwOP5tDKDQWBF7E1.png
paper_authors.fullname: Zhizhong Huang
paper_authors.type: user
paper_authors.user: Hzzone
paper_authors._id: 6711c8f0fe58fd40dde8c613
paper_authors.name: Jiaxin Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T12:43:21.961Z
paper_authors._id: 6503b02993574a89717b8f91
paper_authors.avatarUrl: /avatars/f4d1698f1f7c516c5b849b96351a885e.svg
paper_authors.fullname: JiaxinYe
paper_authors.type: user
paper_authors.user: JiaxinYe
paper_authors._id: 6711c8f0fe58fd40dde8c614
paper_authors.name: Yingya Zhang
paper_authors._id: 6711c8f0fe58fd40dde8c615
paper_authors.name: Hongming Shan
paper_title: DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise
    Motion Control
paper_summary: Recent advances in customized video generation have enabled users to create
  videos tailored to both specific subjects and motion trajectories. However,
  existing methods often require complicated test-time fine-tuning and struggle
  with balancing subject learning and motion control, limiting their real-world
  applications. In this paper, we present DreamVideo-2, a zero-shot video
  customization framework capable of generating videos with a specific subject
  and motion trajectory, guided by a single image and a bounding box sequence,
  respectively, and without the need for test-time fine-tuning. Specifically, we
  introduce reference attention, which leverages the model's inherent
  capabilities for subject learning, and devise a mask-guided motion module to
  achieve precise motion control by fully utilizing the robust motion signal of
  box masks derived from bounding boxes. While these two components achieve their
  intended functions, we empirically observe that motion control tends to
  dominate over subject learning. To address this, we propose two key designs: 1)
  the masked reference attention, which integrates a blended latent mask modeling
  scheme into reference attention to enhance subject representations at the
  desired positions, and 2) a reweighted diffusion loss, which differentiates the
  contributions of regions inside and outside the bounding boxes to ensure a
  balance between subject and motion control. Extensive experimental results on a
  newly curated dataset demonstrate that DreamVideo-2 outperforms
  state-of-the-art methods in both subject customization and motion control. The
  dataset, code, and models will be made publicly available.
paper_ai_summary: DreamVideo-2 generates customized videos using a single image and bounding box sequence without test-time fine-tuning, balancing subject learning and motion control through masked reference attention and reweighted diffusion loss.
paper_ai_keywords: zero-shot video customization reference attention mask-guided motion module blended latent mask modeling reweighted diffusion loss