Reenact Anything: Semantic Video Motion Transfer Using Motion-Textual
  Inversion
Recent years have seen a tremendous improvement in the quality of video
generation and editing approaches. While several techniques focus on editing
appearance, few address motion. Current approaches using text, trajectories, or
bounding boxes are limited to simple motions, so we specify motions with a
single motion reference video instead. We further propose to use a pre-trained
image-to-video model rather than a text-to-video model. This approach allows us
to preserve the exact appearance and position of a target object or scene and
helps disentangle appearance from motion. Our method, called motion-textual
inversion, leverages our observation that image-to-video models extract
appearance mainly from the (latent) image input, while the text/image embedding
injected via cross-attention predominantly controls motion. We thus represent
motion using text/image embedding tokens. By operating on an inflated
motion-text embedding containing multiple text/image embedding tokens per
frame, we achieve a high temporal motion granularity. Once optimized on the
motion reference video, this embedding can be applied to various target images
to generate videos with semantically similar motions. Our approach does not
require spatial alignment between the motion reference video and target image,
generalizes across various domains, and can be applied to various tasks such as
full-body and face reenactment, as well as controlling the motion of inanimate
objects and the camera. We empirically demonstrate the effectiveness of our
method in the semantic video motion transfer task, significantly outperforming
existing methods in this context.
66ac8be13a79a281ea149a55
Manuel Kansy
claimed_verified
2024-08-02T08:52:59.968Z
66863bd107019f3fe48a21ab
/avatars/3297e18e43d40e902b9554a077a34a8a.svg
Manuel Kansy
user
manuelkansy
66ac8be13a79a281ea149a56
Jacek Naruniec
66ac8be13a79a281ea149a57
Christopher Schroers
66ac8be13a79a281ea149a58
Markus Gross
66ac8be13a79a281ea149a59
Romann M. Weber
admin_assigned
2024-08-02T09:22:09.027Z
630f7646197cd3f24e7f8e9f
/avatars/59bbd4ed38277b313051aac78f6808ac.svg
Romann Weber
user
RMW
Reenact Anything: Semantic Video Motion Transfer Using Motion-Textual
  Inversion
Recent years have seen a tremendous improvement in the quality of video
generation and editing approaches. While several techniques focus on editing
appearance, few address motion. Current approaches using text, trajectories, or
bounding boxes are limited to simple motions, so we specify motions with a
single motion reference video instead. We further propose to use a pre-trained
image-to-video model rather than a text-to-video model. This approach allows us
to preserve the exact appearance and position of a target object or scene and
helps disentangle appearance from motion. Our method, called motion-textual
inversion, leverages our observation that image-to-video models extract
appearance mainly from the (latent) image input, while the text/image embedding
injected via cross-attention predominantly controls motion. We thus represent
motion using text/image embedding tokens. By operating on an inflated
motion-text embedding containing multiple text/image embedding tokens per
frame, we achieve a high temporal motion granularity. Once optimized on the
motion reference video, this embedding can be applied to various target images
to generate videos with semantically similar motions. Our approach does not
require spatial alignment between the motion reference video and target image,
generalizes across various domains, and can be applied to various tasks such as
full-body and face reenactment, as well as controlling the motion of inanimate
objects and the camera. We empirically demonstrate the effectiveness of our
method in the semantic video motion transfer task, significantly outperforming
existing methods in this context.
Motion-textual inversion uses a pre-trained image-to-video model and text/image embeddings to transfer motion from a reference video to different images or objects, achieving high-quality and semantically similar motion transfers.
image-to-video model text/image embedding cross-attention motion-text embedding appearance disentanglement motion transfer video reenactment inanimate objects camera control