DynVFX: Augmenting Real Videos with Dynamic Content
We present a method for augmenting real-world videos with newly generated
dynamic content. Given an input video and a simple user-provided text
instruction describing the desired content, our method synthesizes dynamic
objects or complex scene effects that naturally interact with the existing
scene over time. The position, appearance, and motion of the new content are
seamlessly integrated into the original footage while accounting for camera
motion, occlusions, and interactions with other dynamic objects in the scene,
resulting in a cohesive and realistic output video. We achieve this via a
zero-shot, training-free framework that harnesses a pre-trained text-to-video
diffusion transformer to synthesize the new content and a pre-trained Vision
Language Model to envision the augmented scene in detail. Specifically, we
introduce a novel inference-based method that manipulates features within the
attention mechanism, enabling accurate localization and seamless integration of
the new content while preserving the integrity of the original scene. Our
method is fully automated, requiring only a simple user instruction. We
demonstrate its effectiveness on a wide range of edits applied to real-world
videos, encompassing diverse objects and scenarios involving both camera and
object motion.
67a59e5298f41a0460ee5282
Danah Yatim
admin_assigned
2025-02-07T10:37:14.464Z
6301d8324ccccaa23d3864f4
/avatars/148b1b1d1460e26f03a1f2ce0feacf78.svg
Danah Yatim
user
DanahY
67a59e5298f41a0460ee5283
Rafail Fridman
admin_assigned
2025-02-07T10:37:21.549Z
62627f3c02cd5952e013c843
/avatars/1d76689d75d670630b6fa0307309c31f.svg
Rafail Fridman
user
RafailFridman
67a59e5298f41a0460ee5284
Omer Bar-Tal
admin_assigned
2025-02-07T10:37:28.382Z
62e29044a133a252b5cf70b2
/avatars/6d09ddcba9bc47c309150a8d77815891.svg
Omer Bar-Tal
user
omerbartal
67a59e5298f41a0460ee5285
Tali Dekel
admin_assigned
2025-02-07T10:37:34.275Z
631cddec68f7da9ad24f6fc7
/avatars/7d4f1ce805e5889ca6594bd4a93f2583.svg
Tali Dekel
user
talidekel
DynVFX: Augmenting Real Videos with Dynamic Content
We present a method for augmenting real-world videos with newly generated
dynamic content. Given an input video and a simple user-provided text
instruction describing the desired content, our method synthesizes dynamic
objects or complex scene effects that naturally interact with the existing
scene over time. The position, appearance, and motion of the new content are
seamlessly integrated into the original footage while accounting for camera
motion, occlusions, and interactions with other dynamic objects in the scene,
resulting in a cohesive and realistic output video. We achieve this via a
zero-shot, training-free framework that harnesses a pre-trained text-to-video
diffusion transformer to synthesize the new content and a pre-trained Vision
Language Model to envision the augmented scene in detail. Specifically, we
introduce a novel inference-based method that manipulates features within the
attention mechanism, enabling accurate localization and seamless integration of
the new content while preserving the integrity of the original scene. Our
method is fully automated, requiring only a simple user instruction. We
demonstrate its effectiveness on a wide range of edits applied to real-world
videos, encompassing diverse objects and scenarios involving both camera and
object motion.
A zero-shot, training-free method using text-to-video diffusion transformer and Vision Language Model synthesizes and integrates dynamic content into real-world videos based on text instructions.
text-to-video diffusion transformer Vision Language Model attention mechanism