VideoAnydoor: High-fidelity Video Object Insertion with Precise Motion
  Control
Despite significant advancements in video generation, inserting a given
object into videos remains a challenging task. The difficulty lies in
preserving the appearance details of the reference object and accurately
modeling coherent motions at the same time. In this paper, we propose
VideoAnydoor, a zero-shot video object insertion framework with high-fidelity
detail preservation and precise motion control. Starting from a text-to-video
model, we utilize an ID extractor to inject the global identity and leverage a
box sequence to control the overall motion. To preserve the detailed appearance
and meanwhile support fine-grained motion control, we design a pixel warper. It
takes the reference image with arbitrary key-points and the corresponding
key-point trajectories as inputs. It warps the pixel details according to the
trajectories and fuses the warped features with the diffusion U-Net, thus
improving detail preservation and supporting users in manipulating the motion
trajectories. In addition, we propose a training strategy involving both videos
and static images with a reweight reconstruction loss to enhance insertion
quality. VideoAnydoor demonstrates significant superiority over existing
methods and naturally supports various downstream applications (e.g., talking
head generation, video virtual try-on, multi-region editing) without
task-specific fine-tuning.
677752371ab3b33411033089
Yuanpeng Tu
claimed_verified
2025-01-03T14:03:30.344Z
65fd2da40e543c5a84586eb5
/avatars/ebb4e4bda4b025f167fb9fb4099e4cfd.svg
yuanpeng
user
Tuyuanpeng
677752371ab3b3341103308a
Hao Luo
677752371ab3b3341103308b
Xi Chen
claimed_verified
2025-01-05T23:02:56.258Z
644a1b6401e18bf93a6f45c1
https://cdn-avatars.huggingface.co/v1/production/uploads/644a1b6401e18bf93a6f45c1/P0i_CgCrIzOS2tYRlxoE9.png
xichen
user
xichenhku
677752371ab3b3341103308c
Sihui Ji
claimed_verified
2025-06-05T07:30:34.289Z
64775b6ce5ddd7d2a16fadee
/avatars/19af37ffd7626ebc58b387a34a8f98d7.svg
Sihui Ji
user
zjuJish
677752371ab3b3341103308d
Xiang Bai
admin_assigned
2025-01-06T14:01:17.737Z
641790e2f1e86908935d82a0
/avatars/ced7a137c6344c74b7ac0d5c84833fc8.svg
Xiang Bai
user
baixianger
677752371ab3b3341103308e
Hengshuang Zhao
VideoAnydoor: High-fidelity Video Object Insertion with Precise Motion
  Control
Despite significant advancements in video generation, inserting a given
object into videos remains a challenging task. The difficulty lies in
preserving the appearance details of the reference object and accurately
modeling coherent motions at the same time. In this paper, we propose
VideoAnydoor, a zero-shot video object insertion framework with high-fidelity
detail preservation and precise motion control. Starting from a text-to-video
model, we utilize an ID extractor to inject the global identity and leverage a
box sequence to control the overall motion. To preserve the detailed appearance
and meanwhile support fine-grained motion control, we design a pixel warper. It
takes the reference image with arbitrary key-points and the corresponding
key-point trajectories as inputs. It warps the pixel details according to the
trajectories and fuses the warped features with the diffusion U-Net, thus
improving detail preservation and supporting users in manipulating the motion
trajectories. In addition, we propose a training strategy involving both videos
and static images with a reweight reconstruction loss to enhance insertion
quality. VideoAnydoor demonstrates significant superiority over existing
methods and naturally supports various downstream applications (e.g., talking
head generation, video virtual try-on, multi-region editing) without
task-specific fine-tuning.
VideoAnydoor is a zero-shot framework for object insertion in videos that preserves detail and controls motion using a text-to-video model, ID extractor, box sequence, pixel warper, and diffusion U-Net.
ID extractor box sequence pixel warper diffusion U-Net reweight reconstruction loss