title: DynVFX: Augmenting Real Videos with Dynamic Content
summary: We present a method for augmenting real-world videos with newly generated
  dynamic content. Given an input video and a simple user-provided text
  instruction describing the desired content, our method synthesizes dynamic
  objects or complex scene effects that naturally interact with the existing
  scene over time. The position, appearance, and motion of the new content are
  seamlessly integrated into the original footage while accounting for camera
  motion, occlusions, and interactions with other dynamic objects in the scene,
  resulting in a cohesive and realistic output video. We achieve this via a
  zero-shot, training-free framework that harnesses a pre-trained text-to-video
  diffusion transformer to synthesize the new content and a pre-trained Vision
  Language Model to envision the augmented scene in detail. Specifically, we
  introduce a novel inference-based method that manipulates features within the
  attention mechanism, enabling accurate localization and seamless integration of
  the new content while preserving the integrity of the original scene. Our
  method is fully automated, requiring only a simple user instruction. We
  demonstrate its effectiveness on a wide range of edits applied to real-world
  videos, encompassing diverse objects and scenarios involving both camera and
  object motion.
paper_authors._id: 67a59e5298f41a0460ee5282
paper_authors.name: Danah Yatim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:37:14.464Z
paper_authors._id: 6301d8324ccccaa23d3864f4
paper_authors.avatarUrl: /avatars/148b1b1d1460e26f03a1f2ce0feacf78.svg
paper_authors.fullname: Danah Yatim
paper_authors.type: user
paper_authors.user: DanahY
paper_authors._id: 67a59e5298f41a0460ee5283
paper_authors.name: Rafail Fridman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:37:21.549Z
paper_authors._id: 62627f3c02cd5952e013c843
paper_authors.avatarUrl: /avatars/1d76689d75d670630b6fa0307309c31f.svg
paper_authors.fullname: Rafail Fridman
paper_authors.type: user
paper_authors.user: RafailFridman
paper_authors._id: 67a59e5298f41a0460ee5284
paper_authors.name: Omer Bar-Tal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:37:28.382Z
paper_authors._id: 62e29044a133a252b5cf70b2
paper_authors.avatarUrl: /avatars/6d09ddcba9bc47c309150a8d77815891.svg
paper_authors.fullname: Omer Bar-Tal
paper_authors.type: user
paper_authors.user: omerbartal
paper_authors._id: 67a59e5298f41a0460ee5285
paper_authors.name: Tali Dekel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-07T10:37:34.275Z
paper_authors._id: 631cddec68f7da9ad24f6fc7
paper_authors.avatarUrl: /avatars/7d4f1ce805e5889ca6594bd4a93f2583.svg
paper_authors.fullname: Tali Dekel
paper_authors.type: user
paper_authors.user: talidekel
paper_title: DynVFX: Augmenting Real Videos with Dynamic Content
paper_summary: We present a method for augmenting real-world videos with newly generated
  dynamic content. Given an input video and a simple user-provided text
  instruction describing the desired content, our method synthesizes dynamic
  objects or complex scene effects that naturally interact with the existing
  scene over time. The position, appearance, and motion of the new content are
  seamlessly integrated into the original footage while accounting for camera
  motion, occlusions, and interactions with other dynamic objects in the scene,
  resulting in a cohesive and realistic output video. We achieve this via a
  zero-shot, training-free framework that harnesses a pre-trained text-to-video
  diffusion transformer to synthesize the new content and a pre-trained Vision
  Language Model to envision the augmented scene in detail. Specifically, we
  introduce a novel inference-based method that manipulates features within the
  attention mechanism, enabling accurate localization and seamless integration of
  the new content while preserving the integrity of the original scene. Our
  method is fully automated, requiring only a simple user instruction. We
  demonstrate its effectiveness on a wide range of edits applied to real-world
  videos, encompassing diverse objects and scenarios involving both camera and
  object motion.
paper_ai_summary: A zero-shot, training-free method using text-to-video diffusion transformer and Vision Language Model synthesizes and integrates dynamic content into real-world videos based on text instructions.
paper_ai_keywords: text-to-video diffusion transformer Vision Language Model attention mechanism