title: Reenact Anything: Semantic Video Motion Transfer Using Motion-Textual
    Inversion
summary: Recent years have seen a tremendous improvement in the quality of video
  generation and editing approaches. While several techniques focus on editing
  appearance, few address motion. Current approaches using text, trajectories, or
  bounding boxes are limited to simple motions, so we specify motions with a
  single motion reference video instead. We further propose to use a pre-trained
  image-to-video model rather than a text-to-video model. This approach allows us
  to preserve the exact appearance and position of a target object or scene and
  helps disentangle appearance from motion. Our method, called motion-textual
  inversion, leverages our observation that image-to-video models extract
  appearance mainly from the (latent) image input, while the text/image embedding
  injected via cross-attention predominantly controls motion. We thus represent
  motion using text/image embedding tokens. By operating on an inflated
  motion-text embedding containing multiple text/image embedding tokens per
  frame, we achieve a high temporal motion granularity. Once optimized on the
  motion reference video, this embedding can be applied to various target images
  to generate videos with semantically similar motions. Our approach does not
  require spatial alignment between the motion reference video and target image,
  generalizes across various domains, and can be applied to various tasks such as
  full-body and face reenactment, as well as controlling the motion of inanimate
  objects and the camera. We empirically demonstrate the effectiveness of our
  method in the semantic video motion transfer task, significantly outperforming
  existing methods in this context.
paper_authors._id: 66ac8be13a79a281ea149a55
paper_authors.name: Manuel Kansy
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-02T08:52:59.968Z
paper_authors._id: 66863bd107019f3fe48a21ab
paper_authors.avatarUrl: /avatars/3297e18e43d40e902b9554a077a34a8a.svg
paper_authors.fullname: Manuel Kansy
paper_authors.type: user
paper_authors.user: manuelkansy
paper_authors._id: 66ac8be13a79a281ea149a56
paper_authors.name: Jacek Naruniec
paper_authors._id: 66ac8be13a79a281ea149a57
paper_authors.name: Christopher Schroers
paper_authors._id: 66ac8be13a79a281ea149a58
paper_authors.name: Markus Gross
paper_authors._id: 66ac8be13a79a281ea149a59
paper_authors.name: Romann M. Weber
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T09:22:09.027Z
paper_authors._id: 630f7646197cd3f24e7f8e9f
paper_authors.avatarUrl: /avatars/59bbd4ed38277b313051aac78f6808ac.svg
paper_authors.fullname: Romann Weber
paper_authors.type: user
paper_authors.user: RMW
paper_title: Reenact Anything: Semantic Video Motion Transfer Using Motion-Textual
    Inversion
paper_summary: Recent years have seen a tremendous improvement in the quality of video
  generation and editing approaches. While several techniques focus on editing
  appearance, few address motion. Current approaches using text, trajectories, or
  bounding boxes are limited to simple motions, so we specify motions with a
  single motion reference video instead. We further propose to use a pre-trained
  image-to-video model rather than a text-to-video model. This approach allows us
  to preserve the exact appearance and position of a target object or scene and
  helps disentangle appearance from motion. Our method, called motion-textual
  inversion, leverages our observation that image-to-video models extract
  appearance mainly from the (latent) image input, while the text/image embedding
  injected via cross-attention predominantly controls motion. We thus represent
  motion using text/image embedding tokens. By operating on an inflated
  motion-text embedding containing multiple text/image embedding tokens per
  frame, we achieve a high temporal motion granularity. Once optimized on the
  motion reference video, this embedding can be applied to various target images
  to generate videos with semantically similar motions. Our approach does not
  require spatial alignment between the motion reference video and target image,
  generalizes across various domains, and can be applied to various tasks such as
  full-body and face reenactment, as well as controlling the motion of inanimate
  objects and the camera. We empirically demonstrate the effectiveness of our
  method in the semantic video motion transfer task, significantly outperforming
  existing methods in this context.
paper_ai_keywords: motion-textual inversion pre-trained image-to-video model cross-attention text/image embedding tokens inflated motion-text embedding motion reference video semantic video motion transfer