title: VideoAnydoor: High-fidelity Video Object Insertion with Precise Motion
    Control
summary: Despite significant advancements in video generation, inserting a given
  object into videos remains a challenging task. The difficulty lies in
  preserving the appearance details of the reference object and accurately
  modeling coherent motions at the same time. In this paper, we propose
  VideoAnydoor, a zero-shot video object insertion framework with high-fidelity
  detail preservation and precise motion control. Starting from a text-to-video
  model, we utilize an ID extractor to inject the global identity and leverage a
  box sequence to control the overall motion. To preserve the detailed appearance
  and meanwhile support fine-grained motion control, we design a pixel warper. It
  takes the reference image with arbitrary key-points and the corresponding
  key-point trajectories as inputs. It warps the pixel details according to the
  trajectories and fuses the warped features with the diffusion U-Net, thus
  improving detail preservation and supporting users in manipulating the motion
  trajectories. In addition, we propose a training strategy involving both videos
  and static images with a reweight reconstruction loss to enhance insertion
  quality. VideoAnydoor demonstrates significant superiority over existing
  methods and naturally supports various downstream applications (e.g., talking
  head generation, video virtual try-on, multi-region editing) without
  task-specific fine-tuning.
paper_authors._id: 677752371ab3b33411033089
paper_authors.name: Yuanpeng Tu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-03T14:03:30.344Z
paper_authors._id: 65fd2da40e543c5a84586eb5
paper_authors.avatarUrl: /avatars/ebb4e4bda4b025f167fb9fb4099e4cfd.svg
paper_authors.fullname: yuanpeng
paper_authors.type: user
paper_authors.user: Tuyuanpeng
paper_authors._id: 677752371ab3b3341103308a
paper_authors.name: Hao Luo
paper_authors._id: 677752371ab3b3341103308b
paper_authors.name: Xi Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-05T23:02:56.258Z
paper_authors._id: 644a1b6401e18bf93a6f45c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/644a1b6401e18bf93a6f45c1/P0i_CgCrIzOS2tYRlxoE9.png
paper_authors.fullname: xichen
paper_authors.type: user
paper_authors.user: xichenhku
paper_authors._id: 677752371ab3b3341103308c
paper_authors.name: Sihui Ji
paper_authors._id: 677752371ab3b3341103308d
paper_authors.name: Xiang Bai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T14:01:17.737Z
paper_authors._id: 641790e2f1e86908935d82a0
paper_authors.avatarUrl: /avatars/ced7a137c6344c74b7ac0d5c84833fc8.svg
paper_authors.fullname: Xiang Bai
paper_authors.type: user
paper_authors.user: baixianger
paper_authors._id: 677752371ab3b3341103308e
paper_authors.name: Hengshuang Zhao
paper_title: VideoAnydoor: High-fidelity Video Object Insertion with Precise Motion
    Control
paper_summary: Despite significant advancements in video generation, inserting a given
  object into videos remains a challenging task. The difficulty lies in
  preserving the appearance details of the reference object and accurately
  modeling coherent motions at the same time. In this paper, we propose
  VideoAnydoor, a zero-shot video object insertion framework with high-fidelity
  detail preservation and precise motion control. Starting from a text-to-video
  model, we utilize an ID extractor to inject the global identity and leverage a
  box sequence to control the overall motion. To preserve the detailed appearance
  and meanwhile support fine-grained motion control, we design a pixel warper. It
  takes the reference image with arbitrary key-points and the corresponding
  key-point trajectories as inputs. It warps the pixel details according to the
  trajectories and fuses the warped features with the diffusion U-Net, thus
  improving detail preservation and supporting users in manipulating the motion
  trajectories. In addition, we propose a training strategy involving both videos
  and static images with a reweight reconstruction loss to enhance insertion
  quality. VideoAnydoor demonstrates significant superiority over existing
  methods and naturally supports various downstream applications (e.g., talking
  head generation, video virtual try-on, multi-region editing) without
  task-specific fine-tuning.
paper_ai_keywords: ID extractor pixel warper diffusion U-Net reweight reconstruction loss talking head generation video virtual try-on multi-region editing