title: Puppet-Master: Scaling Interactive Video Generation as a Motion Prior
    for Part-Level Dynamics
summary: We present Puppet-Master, an interactive video generative model that can
  serve as a motion prior for part-level dynamics. At test time, given a single
  image and a sparse set of motion trajectories (i.e., drags), Puppet-Master can
  synthesize a video depicting realistic part-level motion faithful to the given
  drag interactions. This is achieved by fine-tuning a large-scale pre-trained
  video diffusion model, for which we propose a new conditioning architecture to
  inject the dragging control effectively. More importantly, we introduce the
  all-to-first attention mechanism, a drop-in replacement for the widely adopted
  spatial attention modules, which significantly improves generation quality by
  addressing the appearance and background issues in existing models. Unlike
  other motion-conditioned video generators that are trained on in-the-wild
  videos and mostly move an entire object, Puppet-Master is learned from
  Objaverse-Animation-HQ, a new dataset of curated part-level motion clips. We
  propose a strategy to automatically filter out sub-optimal animations and
  augment the synthetic renderings with meaningful motion trajectories.
  Puppet-Master generalizes well to real images across various categories and
  outperforms existing methods in a zero-shot manner on a real-world benchmark.
  See our project page for more results: vgg-puppetmaster.github.io.
paper_authors._id: 66b5756218d95c926cb7132e
paper_authors.name: Ruining Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-09T10:22:06.290Z
paper_authors._id: 6467cc17e7a6a374fd1a41e5
paper_authors.avatarUrl: /avatars/3cbd4ad5bc17067ee38b13d0cbadd80c.svg
paper_authors.fullname: Ruining Li
paper_authors.type: user
paper_authors.user: rayli
paper_authors._id: 66b5756218d95c926cb7132f
paper_authors.name: Chuanxia Zheng
paper_authors._id: 66b5756218d95c926cb71330
paper_authors.name: Christian Rupprecht
paper_authors._id: 66b5756218d95c926cb71331
paper_authors.name: Andrea Vedaldi
paper_title: Puppet-Master: Scaling Interactive Video Generation as a Motion Prior
    for Part-Level Dynamics
paper_summary: We present Puppet-Master, an interactive video generative model that can
  serve as a motion prior for part-level dynamics. At test time, given a single
  image and a sparse set of motion trajectories (i.e., drags), Puppet-Master can
  synthesize a video depicting realistic part-level motion faithful to the given
  drag interactions. This is achieved by fine-tuning a large-scale pre-trained
  video diffusion model, for which we propose a new conditioning architecture to
  inject the dragging control effectively. More importantly, we introduce the
  all-to-first attention mechanism, a drop-in replacement for the widely adopted
  spatial attention modules, which significantly improves generation quality by
  addressing the appearance and background issues in existing models. Unlike
  other motion-conditioned video generators that are trained on in-the-wild
  videos and mostly move an entire object, Puppet-Master is learned from
  Objaverse-Animation-HQ, a new dataset of curated part-level motion clips. We
  propose a strategy to automatically filter out sub-optimal animations and
  augment the synthetic renderings with meaningful motion trajectories.
  Puppet-Master generalizes well to real images across various categories and
  outperforms existing methods in a zero-shot manner on a real-world benchmark.
  See our project page for more results: vgg-puppetmaster.github.io.
paper_ai_keywords: video generative model motion prior part-level dynamics video diffusion model conditioning architecture all-to-first attention mechanism spatial attention modules Objaverse-Animation-HQ part-level motion clips zero-shot manner