title: DragNUWA: Fine-grained Control in Video Generation by Integrating Text,
    Image, and Trajectory
summary: Controllable video generation has gained significant attention in recent
  years. However, two main limitations persist: Firstly, most existing works
  focus on either text, image, or trajectory-based control, leading to an
  inability to achieve fine-grained control in videos. Secondly, trajectory
  control research is still in its early stages, with most experiments being
  conducted on simple datasets like Human3.6M. This constraint limits the models'
  capability to process open-domain images and effectively handle complex curved
  trajectories. In this paper, we propose DragNUWA, an open-domain
  diffusion-based video generation model. To tackle the issue of insufficient
  control granularity in existing works, we simultaneously introduce text, image,
  and trajectory information to provide fine-grained control over video content
  from semantic, spatial, and temporal perspectives. To resolve the problem of
  limited open-domain trajectory control in current research, We propose
  trajectory modeling with three aspects: a Trajectory Sampler (TS) to enable
  open-domain control of arbitrary trajectories, a Multiscale Fusion (MF) to
  control trajectories in different granularities, and an Adaptive Training (AT)
  strategy to generate consistent videos following trajectories. Our experiments
  validate the effectiveness of DragNUWA, demonstrating its superior performance
  in fine-grained control in video generation. The homepage link is
  https://www.microsoft.com/en-us/research/project/dragnuwa/
paper_authors._id: 64ddbe671a14509152994239
paper_authors.name: Shengming Yin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-17T14:11:05.764Z
paper_authors._id: 62430b00b1fc6718f4fa6295
paper_authors.avatarUrl: /avatars/f4554c124ec12ff0457598b5db4b175b.svg
paper_authors.fullname: LanHarmony
paper_authors.type: user
paper_authors.user: LanHarmony
paper_authors._id: 64ddbe671a1450915299423a
paper_authors.name: Chenfei Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-17T14:11:18.425Z
paper_authors._id: 63180f0f49af4d408c35f07f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678864054390-63180f0f49af4d408c35f07f.jpeg
paper_authors.fullname: Chenfei Wu
paper_authors.type: user
paper_authors.user: Chenfei
paper_authors._id: 64ddbe671a1450915299423b
paper_authors.name: Jian Liang
paper_authors._id: 64ddbe671a1450915299423c
paper_authors.name: Jie Shi
paper_authors._id: 64ddbe671a1450915299423d
paper_authors.name: Houqiang Li
paper_authors._id: 64ddbe671a1450915299423e
paper_authors.name: Gong Ming
paper_authors._id: 64ddbe671a1450915299423f
paper_authors.name: Nan Duan
paper_title: DragNUWA: Fine-grained Control in Video Generation by Integrating Text,
    Image, and Trajectory
paper_summary: Controllable video generation has gained significant attention in recent
  years. However, two main limitations persist: Firstly, most existing works
  focus on either text, image, or trajectory-based control, leading to an
  inability to achieve fine-grained control in videos. Secondly, trajectory
  control research is still in its early stages, with most experiments being
  conducted on simple datasets like Human3.6M. This constraint limits the models'
  capability to process open-domain images and effectively handle complex curved
  trajectories. In this paper, we propose DragNUWA, an open-domain
  diffusion-based video generation model. To tackle the issue of insufficient
  control granularity in existing works, we simultaneously introduce text, image,
  and trajectory information to provide fine-grained control over video content
  from semantic, spatial, and temporal perspectives. To resolve the problem of
  limited open-domain trajectory control in current research, We propose
  trajectory modeling with three aspects: a Trajectory Sampler (TS) to enable
  open-domain control of arbitrary trajectories, a Multiscale Fusion (MF) to
  control trajectories in different granularities, and an Adaptive Training (AT)
  strategy to generate consistent videos following trajectories. Our experiments
  validate the effectiveness of DragNUWA, demonstrating its superior performance
  in fine-grained control in video generation. The homepage link is
  https://www.microsoft.com/en-us/research/project/dragnuwa/
paper_ai_keywords: diffusion-based video generation fine-grained control text-based control image-based control trajectory-based control Trajectory Sampler (TS) Multiscale Fusion (MF) Adaptive Training (AT) Human3.6M open-domain images complex curved trajectories consistent videos