## Reenact Anything: Semantic Video Motion Transfer Using Motion-Textual Inversion

MANUEL KANSY, ETH Zürich &amp; DisneyResearch|Studios, Switzerland JACEK NARUNIEC, DisneyResearch|Studios, Switzerland CHRISTOPHER SCHROERS, DisneyResearch|Studios, Switzerland MARKUS GROSS, ETH Zürich &amp; DisneyResearch|Studios, Switzerland ROMANN M. WEBER, DisneyResearch|Studios, Switzerland

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00458v1-figure-a3390ea7-ab78-47a6-b8a1-cd07e04eb1f0.jpg)

Fig. 1. We encode the motion of a reference video into a novel motion-text embedding using a frozen, pre-trained image-to-video diffusion model. This optimized motion-text embedding can then be applied to different starting images to generate videos with semantically similar motions. The general nature of our motion representation allows for successful motion transfer even when objects are not spatially aligned, across various domains, and for multiple objects. Additionally, our method supports multiple types of motions, including full-body, face, camera, and even hand-crafted motions.

All videos in this paper are best viewed as animations with Acrobat Reader by pressing the highlighted frame of each video.

Recent years have seen a tremendous improvement in the quality of video generation and editing approaches. While several techniques focus on editing appearance, few address motion. Current approaches using text, trajectories, or bounding boxes are limited to simple motions, so we specify motions with a single motion reference video instead. We further propose to use a pre-trained image-to-video model rather than a text-to-video model. This approach allows us to preserve the exact appearance and position of a target object or scene and helps disentangle appearance from motion.

image input, while the text/image embedding injected via cross-attention predominantly controls motion. We thus represent motion using text/image embedding tokens. By operating on an inflated motion-text embedding containing multiple text/image embedding tokens per frame, we achieve a high temporal motion granularity. Once optimized on the motion reference video, this embedding can be applied to various target images to generate videos with semantically similar motions.

Our method, called motion-textual inversion , leverages our observation that image-to-video models extract appearance mainly from the (latent)

Authors' addresses: Manuel Kansy, ETH Zürich &amp; DisneyResearch|Studios, Zürich, Switzerland, manuel.kansy@inf.ethz.ch; Jacek Naruniec, DisneyResearch|Studios, Zürich, Switzerland, jacek.naruniec@disneyresearch.com; Christopher Schroers, DisneyResearch|Studios, Zürich, Switzerland, christopher.schroers@disneyresearch.com; Markus Gross, ETH Zürich &amp; DisneyResearch|Studios, Zürich, Switzerland, christopher. schroers@disneyresearch.com; Romann M. Weber, DisneyResearch|Studios, Zürich, Switzerland, romann.weber@disneyresearch.com.

Our approach does not require spatial alignment between the motion reference video and target image, generalizes across various domains, and can be applied to various tasks such as full-body and face reenactment, as well as controlling the motion of inanimate objects and the camera. We empirically demonstrate the effectiveness of our method in the semantic video motion transfer task, significantly outperforming existing methods in this context.

## 1 INTRODUCTION

Since the advent of diffusion models, video generation and editing methods have significantly improved. However, controlling the motions generated by these models remains challenging. For instance, Stable Video Diffusion [Blattmann et al. 2023a], a state-of-the-art image-to-video diffusion model, offers little practical control over motion. The primary methods to modify motion involve altering the diffusion process's random seed or adjusting micro-conditioning inputs like frame rate, but these options are not easily interpretable 1 .

Existing methods for controlling motion with sparse control signals like text [Dai et al. 2023; Li et al. 2024a; Molad et al. 2023; Yan et al. 2023], boxes [Chen et al. 2024; Jain et al. 2023; Li et al. 2024a; Ma et al. 2023; Wang et al. 2024c], or trajectories [Chen et al. 2023a; Li et al. 2024b; Mou et al. 2024; Niu et al. 2024; Qiu et al. 2024; Wu et al. 2024a; Yin et al. 2023] are limited to simple motions in most practical scenarios. On the other hand, dense motion trajectories [Wang et al. 2024b] may leak the motion reference video's spatial structure, thus often failing in unaligned scenarios.

Many reference-based motion transfer methods fine-tune model components (e.g., using LoRA [Hu et al. 2021]) on one or several motion reference videos [Jeong et al. 2023; Materzynska et al. 2023; Wei et al. 2023; Wu et al. 2023b,a; Zhang et al. 2023a; Zhao et al. 2023a]. This often results in appearance leakage, where the model overfits to the reference video's appearance and fails to generalize to new target object appearances. Motion transfer techniques that use the inversion-then-generation framework [Bai et al. 2024; Ling et al. 2024; Yatim et al. 2023] attempt to replicate the reference video's structure in the generated video. However, this approach can be problematic when there are significant differences between the locations and geometries of the reference and target objects, leading to misaligned semantic features.

Most of the one-shot reference-based methods produce videos with motions that are mostly spatially aligned with the motion reference video, i.e., they follow the layout as well as the subject scale and position of the reference video. We thus argue that many of these works [Jeong et al. 2023; Ling et al. 2024; Yatim et al. 2023; Zhang et al. 2023a] can be considered as an advanced form of appearance transfer rather than motion transfer. Lastly, most existing methods are based on text-to-video models and thus cannot preserve the exact appearance and background of a given target image.

Wepropose to transfer the semantic motion of a motion reference video to a given target image. Specifically, given motion reference video 𝑉 𝑅 and target image 𝐼 𝑇 , we want to generate a video ˆ 𝑉 such that 𝑀 sem ( ˆ 𝑉 ) = 𝑀 sem ( 𝑉 𝑅 ) , where 𝑀 sem is the semantic motion, and 𝐴 𝑉 ( ˆ ) = 𝐴 𝐼 ( 𝑇 ) , where 𝐴 is the appearance and spatial layout. Importantly, the generated motion should match the semantics of the motion reference video rather than the exact same spatial motions. For example, we want to be able to generate a video of a given subject doing jumping jacks on the left side of the video even if the subject in the motion reference video is in the center. Fig. 1 shows exemplary results of our method, including motion transfers to multiple (misaligned) objects.

1 The authors demonstrate camera motion control by fine-tuning motion-specific LoRA [Hu et al. 2021] modules; however, we consider this an extension rather than a feature of the base model.

We identify two key challenges: appearance leakage from the motion reference video and object misalignment. To tackle appearance leakage, we employ an image-to-video rather than a text-tovideo (or an inflated text-to-image) model and do not fine-tune the model. To the best of our knowledge, we are the first to use an image-to-video model for general motion transfer. To address object misalignments between the motion reference video and the target image, we introduce a novel motion representation that eliminates the need for spatial alignment by not having a spatial dimension in the first place.

Our motion representation is based on our observation that imageto-video models extract the appearance predominantly from the (latent) image input, whereas the text/image embedding injected via cross-attention mostly controls the motion. We therefore propose to represent motion with several text/image embedding tokens, together referred to as motion-text embedding , that we optimize on a given motion reference video. Thereby, our inflated motiontext embedding enables us to preserve the timing of the motion video very precisely, which is crucial for applications such as visual dubbing. Our approach, named motion-textual inversion , is general in nature and works for various types of motions and objects. Perhaps surprising at first, it turns out that while words are not ideal for describing motions, their embeddings can describe highly complex motions exceptionally well, as shown in our experiments.

To summarize, our contributions are:

- (1) We introduce the semantic video motion transfer task in an image-to-video setting.
- (2) We observe that text/image embeddings of image-to-video diffusion models store and affect motion and leverage them as a general and compact motion representation.
- (3) We propose motion-textual inversion , a novel method that optimizes multiple text/image embedding tokens on a motion reference video and transfers the learned motion to target images.
- (4) We demonstrate superior performance over existing motion transfer approaches.

## 2 RELATED WORK

## 2.1 Domain-Specific Reenactment

Reenactment has been a significant research area, but much of the focus has been on domain-specific approaches like face reenactment [Drobyshev et al. 2022; Guo et al. 2024; Hsu et al. 2022; Li et al. 2023; Nirkin et al. 2019; Wang et al. 2021] and human full-body motion transfer [Chan et al. 2019; Hu 2024; Ma et al. 2024; Tu et al. 2024; Zhu et al. 2024; Zuo et al. 2024]. While these methods perform well, their architectures and training data are tailored to specific domains, making it challenging to adapt them for use across multiple domains. Current general image animation methods [Siarohin et al. 2019, 2021; Zhao and Zhang 2022] also still require domain-specific training, so they cannot directly solve our task.

Since our aim was to introduce a general method, our design choices differ from domain-specific models. For instance, many reenactment techniques rely on keypoints, which are derived either from pre-trained, domain-specific landmark detectors [Chan et al. 2019; Hsu et al. 2022; Hu 2024; Ma et al. 2024; Nirkin et al.

2019; Tu et al. 2024; Zuo et al. 2024] or learned in an unsupervised manner [Drobyshev et al. 2022; Guo et al. 2024; Siarohin et al. 2019, 2021; Wang et al. 2021; Zhao and Zhang 2022]. Although the latter approach is more flexible, it still requires a separate training for each domain, which is cumbersome. Additionally, this approach presents challenges in determining keypoint placement for unseen domains during inference (e.g., inanimate objects), cross-domain transfers (e.g., human-to-animal), and misaligned objects. We therefore choose an implicit motion representation rather than an explicit one.

Given the impressive cross-domain translation capabilities of diffusion models, as demonstrated in works like [Hertz et al. 2022; Parmar et al. 2023; Tumanyan et al. 2023], we utilize a diffusionbased video foundation model for our general task to take advantage of its extensive and general priors.

## 2.2 Video Generation

Following the rise of text-to-image diffusion models [Ramesh et al. 2022; Rombach et al. 2022; Saharia et al. 2022], video generation models have also greatly improved in quality in recent years. Many text-to-video methods start with a pre-trained text-to-image model and inflate it by adding and training temporal convolution and attention blocks after each corresponding spatial block [Bar-Tal et al. 2024; Blattmann et al. 2023b; Guo et al. 2023; Wang et al. 2023b]. Similarly, many image-to-video diffusion models use a pre-trained text-to-image [Zhang et al. 2023b] or text-to-video [Blattmann et al. 2023a] model as a starting point. They then adapt the model to the image-to-video task by conditioning the model on the image, e.g., by adding [Zhang et al. 2023b] or concatenating [Blattmann et al. 2023a] it to the noisy input. The text embedding input from the pretrained model is either kept [Zhang et al. 2023b] or replaced with an image embedding input [Blattmann et al. 2023a]. While training a custom video generation model provides the most freedom in terms of design choices, it is very expensive in terms of computation and data. Even fine-tuning video models requires substantial resources, so we decided to use a pre-trained diffusion model, Stable Video Diffusion [Blattmann et al. 2023a], and keep it frozen. Additionally, we aim for our method to be applicable to a wide range of motions and subjects. In contrast, approaches that involve training the model often focus on a single type of motion, such as human full-body motion [Hu 2024; Ma et al. 2024].

## 2.3 Video Motion Editing with Explicit Motions

2.3.1 Based on Sparse Control Signals. In theory, the motion of all video generation models that have a text input can simply be controlled by text [Dai et al. 2023; Li et al. 2024a; Molad et al. 2023; Yan et al. 2023], but this approach struggles with complex motions in practice. For more precise spatial control, recent methods use bounding boxes, either with training [Li et al. 2024a; Wang et al. 2024c] or without [Chen et al. 2024; Jain et al. 2023; Ma et al. 2023], and trajectories [Chen et al. 2023a; Li et al. 2024b; Mou et al. 2024; Niu et al. 2024; Qiu et al. 2024; Wu et al. 2024a; Yin et al. 2023], but they rely on consistent spatial alignment for effective motion transfer. Similarly, keypoints are another option for describing motions [Gu et al. 2023; Niu et al. 2024; Tanveer et al. 2024], but they suffer from

•

3

the challenges outlined in Section 2.1. Additionally, some methods focus specifically on camera motions [Bahmani et al. 2024; He et al. 2024; Hou et al. 2024; Hu et al. 2024; Xu et al. 2024] or combine camera and bounding box motions [Wang et al. 2023c; Wu et al. 2024b; Yang et al. 2024]. However, all these approaches are either limited to simple motions or require significant effort to specify complex ones. For instance, a bounding box can specify an object's location (e.g., a person) but not the detailed motion within it (e.g., doing jumping jacks), and modeling complex motion with part-based boxes quickly becomes impractical.

2.3.2 Based on Dense Control Signals. Dense control signals, such as motion vectors [Wang et al. 2024b] and depth maps [Chen et al. 2023b; Wang et al. 2024b; Zhang et al. 2023c], allow for a more precise motion specification. However, using them for general motion transfer is challenging because they also encode information about image and object structure. This can result in unnatural motions when there is a mismatch between the structures of the target image and the reference video as shown in [Wang et al. 2023c].

## 2.4 Video Motion Editing with Implicit Motions

This subsection covers methods for implicitly representing and transferring motion from a reference video. Fine-tuning approaches store motion in model weights, whereas inversion-then-generation methods store motion in model features and attention maps. Some techniques combine both paradigms. When the layout of the subjects in the reference and generated videos match, a given transfer can be seen as either changing the appearance to match the target image or altering the motion to match the reference video. Our focus is on motion transfer where the layouts do not align, a less explored area in the literature, as discussed in Section 2.4.3.

2.4.1 Fine-Tuning. Many fine-tuning methods are inspired by image customization techniques like DreamBooth [Ruiz et al. 2023] and LoRA [Hu et al. 2021]. Loosely speaking, the idea is to fine-tune the parts of the model responsible for motion but avoid training the parts responsible for appearance. Tune-A-Video [Wu et al. 2023b] inflates a text-to-image model by adding spatio-temporal attention and only trains some parts of the attention layers. Similarly, [Materzynska et al. 2023] only fine-tunes parts of the model and further focuses the training more on earlier denoising steps to emphasize learning the general motion rather than fine appearance details. MotionDirector [Zhao et al. 2023a] proposes a dual-path LoRA architecture and an appearance-debiased temporal loss to disentangle appearance from motion. Similarly, DreamVideo [Wei et al. 2023], MotionCrafter [Zhang et al. 2023a], and Customize-AVideo [Ren et al. 2024] have separate branches for appearance and motion. VMC [Jeong et al. 2023] adapts temporal attention layers using a motion distillation strategy with residual vectors between consecutive noisy latent frames as the motion reference.

Fine-tuning a model carries the risk of appearance leakage, which is why many of the aforementioned methods focus on preventing it. We find that using an image-to-video model instead of a text-tovideo model largely avoids these problems. LAMP [Wu et al. 2023a] is the most similar method to ours in that sense, but they adapt a pre-trained text-to-image model to the image-to-video task and

fine-tune it only briefly. In contrast, we employ a pre-trained, largescale image-to-video model to leverage stronger priors for better generalization.

2.4.2 Inversion-then-Generation. The inversion-then-generation framework, initially developed for image editing [Hertz et al. 2022; Parmar et al. 2023; Tumanyan et al. 2023], involves first inverting a reference video into 'noise' using methods like DDIM [Song et al. 2020a] to enable reconstruction through backward diffusion. Thereby, features such as self-attention maps are extracted from the reference video and then injected into the diffusion process of the video being generated. These features either directly replace existing features [Tumanyan et al. 2023] or are incorporated into a loss function [Parmar et al. 2023], ensuring the generated video has a similar structure. Numerous methods have been proposed within this framework for video appearance editing [Bai et al. 2024; Ceylan et al. 2023; Geyer et al. 2023; Harsha et al. 2024; Liu et al. 2023; Wang et al. 2023a; Yang et al. 2023; Zhao et al. 2023b] and video motion editing [Bai et al. 2024; Ling et al. 2024; Yatim et al. 2023], mainly differing in their inversion techniques and feature choices.

The methods mentioned above face several inherent issues in motion transfer tasks. Most notably, they often assume or enforce that the features of the reference and target videos are identical, which leads to problems when generating videos with different geometries or spatial layouts. Some methods attempt to address this by collapsing the spatial dimension of features before using them in a loss [Yatim et al. 2023], but they still typically produce motions with similar directions in pixel space. This limits control and diversity and can produce less natural results. Furthermore, these approaches require tuning numerous hyperparameters (choice of feature, layers, time steps) and necessitate inverting the video, which is challenging for high guidance scales [Mokady et al. 2023] and when using few time steps [Garibi et al. 2024].

2.4.3 With Different Spatial Layout. To avoid being restricted to the layout of a single motion reference video, some methods use multiple motion videos [Materzynska et al. 2023; Wei et al. 2023; Wu et al. 2023a; Zhao et al. 2023a]. However, our goal is to transfer motion with precise temporal alignment to the reference video. This would require multiple temporally-aligned videos, which are often impractical to obtain. Additionally, many motion editing methods with spatial variations [Materzynska et al. 2023; Ren et al. 2024; Wanget al. 2024a] use text to define the subject's appearance instead of an image, resulting in videos that only roughly match the input image. The approach in [Wang et al. 2024a] is most similar to ours as it keeps the model frozen and learns a motion embedding like we do, but it also suffers from the above limitation.

## 3 METHOD

We propose to transfer the semantic motion of a motion reference video to a given target image by motion-textual inversion . We thereby optimize a set of text/image embedding tokens, which we refer to as motion-text embedding , for the motion reference video using a pre-trained image-to-video diffusion model.

## 3.1 Preliminaries

Diffusion models [Ho et al. 2020; Song et al. 2020b] consist of two processes. In the forward process , Gaussian noise is iteratively added to a clean data sample x 0 until it is approximately pure noise. In the reverse process , starting with pure noise x 𝑇 , a learnable denoiser 𝐷 𝜃 iteratively removes noise to obtain a sample that matches the original data distribution 𝑝 data . We follow the continuous-time framework [Karras et al. 2022; Song et al. 2020b], where the denoiser is trained via denoising score matching :

$$\underset { \omega } { m } \underset { \omega } { \math$$

where x 0 is a clean data sample and c an arbitrary conditioning signal from the original data distribution 𝑝 data ; 𝑝 𝜎, ( n ) = 𝑝 𝜎 ( )N( n 0 ; , 𝜎 2 ) , where 𝑝 𝜎 ( ) is a probability distribution over noise levels 𝜎 , and n is noise; and 𝜆 𝜎 : R + → R + is a weighting function. The denoiser 𝐷 𝜃 is parameterized as

$$D _ { \theta } ( { \mathbf x } ; \sigma ) = c _$$

where 𝐹 𝜃 is the neural network to be trained; 𝑐 skip ( 𝜎 ) modulates the skip connection; 𝑐 out and 𝑐 in scale the output and input magnitudes respectively; and 𝑐 noise maps noise level 𝜎 into a conditioning input for 𝐹 𝜃 . For more details, please refer to [Karras et al. 2022].

Latent diffusion models [Rombach et al. 2022] operate in the latent space rather than in pixel space to reduce computation and thus enable higher resolutions. First, an encoder E produces a compressed latent z = E( x ) . Then, we perform the diffusion process over z . Lastly, a decoder D reconstructs the latent features back into pixel space. 2

Stable Video Diffusion (SVD) [Blattmann et al. 2023a] is a video latent diffusion model trained in three stages: 1. A text-to-image model [Rombach et al. 2022] is trained or fine-tuned on (image, text) pairs. 2. The diffusion model is inflated by inserting temporal convolution and attention layers following [Blattmann et al. 2023b] and then trained on (video, text) pairs. 3. The diffusion model is refined on a smaller subset of high-quality videos with exact model adaptations and inputs depending on the task (text-to-video, image-to-video, frame interpolation, multi-view generation). For image-to-video generation, the task is to produce a video given its starting frame. The starting frame is supplied to the model in two places: as a CLIP [Radford et al. 2021] image embedding via crossattention (replacing the CLIP text embedding from the text-to-video pre-training) and as a latent repeated across frames and concatenated channel-wise to the video input. Additionally, the model is micro-conditioned on the frame rate, motion amount, and strength of the noise augmentation (applied to first frame latent).

## 3.2 Motivation

Transferring the motion of a reference video to a given target poses two key challenges, which our design solves quite naturally.

- 3.2.1 Challenge 1: Appearance Leakage. Whenfine-tuning a text-tovideo model to learn the motion from a single reference video, there is a risk of overfitting to the reference video's appearance, which can prevent the model from producing the correct target appearance

2 To maintain consistency in notation, we use x for the diagrams and method description, even though the diffusion process actually occurs in latent space.

Fig. 2. Observation 1. In image-to-video models, the image input primarily dictates the appearance of the generated videos. For example, I2VGen-XL [Zhang et al. 2023b] generates a video of a predominantly white horse from a white horse image, even when the input text specifies the horse's color as 'pink.'

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00458v1-figure-41cbfb54-ce77-489d-b844-cacd909cfd41.jpg)

Fig. 3. Observation 2. In image-to-video models, text/image embeddings significantly influence the generated motions. Swapping the CLIP [Radford et al. 2021] image embeddings of a real horse and a toy horse in Stable Video Diffusion [Blattmann et al. 2023a] results in a swap of the motions in the output videos. This suggests that the real horse's embedding encodes a walking motion, while the toy horse's embedding encodes camera motion without object movement.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00458v1-figure-a6cc32f1-7b40-4f94-bebe-258e582b4982.jpg)

during inference. We demonstrate that using a frozen image-tovideo model can preserve the target appearance without any of the special mechanisms from literature described in Section 2.4.1.

Bydesign, image-to-video models generate videos given a starting frame, so generated frames tend to preserve the appearance of the input image. We observe that image-to-video models primarily derive the appearance from the image (latent) input, even with an additional text input, as shown in Fig. 2. This is likely because the model can directly copy (latent) pixels from the first frame instead of hallucinating them from the sparse text input. This strong reliance on the image input reduces the chance of the reference video's appearance leaking through. To further minimize the risk of appearance leakage, we keep the model's weights frozen, so they cannot possibly store the reference video appearance. This also helps retain the rich video understanding and generalization capabilities of the pre-trained model.

We forgo the alignment problem by representing motions with text or image embedding tokens that do not have a spatial dimension in the first place. Our novel motion representation was motivated by the observation shown in Fig. 3. While SVD generated walking motions for an image of a real horse, it generated no object but mostly camera motion for an image of a pink toy horse 3 , perhaps because the model learned that toys do not move. Recall that SVD has the first frame as input in two places: as image latent and as CLIP [Radford et al. 2021] image embedding. When using the image latent of the real horse but the CLIP embedding of the toy horse, the horse in the generated video does not move. Inversely, the toy horse starts walking when using the CLIP embedding of the real horse, implying that the CLIP embedding affects the motion. We believe that these embeddings are not just affecting the motion but are actually the main origin of the motion.

3.2.2 Challenge 2: Handling Object Misalignment. Our goal is to generate videos where subjects perform the same semantic actions, even if they are in different spatial locations or orientations. Handling misaligned objects is especially important when using imageto-video models because the subject's position is determined by the input image, which typically does not match the position in the motion reference video.

Asdiscussed in Section 2.4.2, existing methods using the inversionthen-generation framework inject features from the motion reference video into the generated video, making it closely follow the reference structure. Arguably, these methods do not copy the motion at its origin but rather the per-frame structure that results from a motion (e.g., rough object positions). For the general, unaligned case, these features would first need to be aligned spatially to avoid injecting the structure in the wrong place. This alignment is challenging since the final positions in the generated video are unknown during the diffusion process as they depend on the motion.

Our intuition for why the text/image embeddings determine the motion (which may be surprising at first) is as follows: Videos can be divided into appearance and motion. Appearance is tied to the spatial arrangement of pixels, making it easier to extract it from spatial inputs like image latents. Motion depends on how pixels change over time, requiring a more global, semantic understanding. Thus, it is more effective to modify motion using image embeddings, which contain more semantic information, have no spatial dimension, and are injected in multiple places of the model. Furthermore, SVD was initially trained as a text-to-video model, with CLIP text embeddings describing motions like 'standing', 'walking', or 'running,' incentivizing the model to control motion through cross-attention inputs to effectively denoise training videos.

## 3.3 Motion-Textual Inversion

While using embeddings from different images can alter the generated motion, it does not transfer the motion robustly. Moreover, selecting a specific frame to define a desired motion is difficult since

3 Image was generated using [Tumanyan et al. 2023].

̂

Fig. 4. Method overview. The baseline image-to-video diffusion model, Stable Video Diffusion [Blattmann et al. 2023a] in our case, inputs the first frame in two places: as image (latent) concatenated with the noisy video and as image embedding (some other image-to-video diffusion models may input text embeddings here instead). We propose to replace the image embedding (shown in red in the inference block) with a learned motion-text embedding m ∗ (green). The motion-text embedding m ∗ is optimized directly with a regular diffusion model loss on one given motion reference video x 0 while keeping the diffusion model frozen. For best results, the motion-text embedding is inflated prior to optimization to ( 𝐹 + 1 ) × 𝑁 tokens, where 𝐹 is the number of frames and 𝑁 is a hyperparameter, while keeping the embedding dimension 𝑑 the same to stay compatible with the pre-trained diffusion model. Note that the diffusion process operates in latent space in practice, and other conditionings and model parameterizations [Karras et al. 2022] are omitted for clarity.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00458v1-figure-a413195b-61aa-4d7b-8664-8207ca6c08be.jpg)

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00458v1-figure-7f3057ab-9012-4325-a4fc-dbc5e44aab94.jpg)

motion is rarely captured by a single frame. To address this, we propose optimizing the embedding based on a given motion reference video, which bears some resemblance to textual inversion [Gal et al. 2022]. In analogy to textual inversion, we name our method motiontextual inversion 4 . Note, however, that our method has a completely different goal: using embeddings to encode video motion rather than image appearance.

Fig. 4 shows a high-level overview of our method. Given a single motion reference video x 0 containing 𝐹 frames, we optimize the motion-text embedding m directly by minimizing the diffusion model loss from Equation 1, keeping the diffusion model frozen:

$$\begin{array} {$$

where c encompasses all remaining conditionings of SVD (e.g., first frame latent, time/noise step, and micro-conditionings). All other symbols are defined in Equations 1 and 2.

## 3.4 Motion-Text Embedding and Cross-Attention Inflation

Cross-attention allows the model to dynamically attend to different tokens ( ∼ words in text-to-image and text-to-video) depending on the current features or context. It is computed as follows:

$$\text{Attention} ( Q, K, V ) = M V = \text{softmax$$

where 𝑄 𝐾 𝑉 , , are the queries, keys, and values respectively; 𝑀 is the attention map; 𝑑 𝑎 is the dimension used in the attention operation; 𝜑 𝑖 ( z 𝑡 ) is an intermediate representation of the level 𝑖 features with 𝐶 𝑖 channels; m is the motion-text embedding (or text/image embedding e in case of baseline SVD) with embedding dimension 𝑑 ;

4 In our implementation, it is actually an image embedding, but we refer to it as 'motiontextual inversion' since SVD's image and text embeddings share the same CLIP space, and other I2V methods use text embeddings instead. Also, it feels more intuitive to represent motions as text rather than an image.

and 𝑊 𝑄,𝑖 ∈ R 𝐶 𝑖 × 𝑑 𝑎 , 𝑊 𝐾,𝑖 ∈ R 𝑑 × 𝑑 𝑎 , and 𝑊 𝑉,𝑖 ∈ R 𝑑 × 𝑑 𝑎 are learned weight matrices for queries, keys, and values respectively.

The image embedding of SVD only has one token. This results in a degenerate case for the cross-attention operation where all entries of the attention map 𝑀 are 1, as shown in Fig. 5a. The model thus attends 100% to that single token and applies its value to all spatial and temporal locations.

3.4.1 Multiple Tokens. Using the same value 𝑉 for all locations limits the extent to which the embedding can change the motion. We therefore propose to have 𝑁 tokens (instead of 1) to attend to, recovering the scenario from the text-to-image or text-to-video pretraining. This allows the model to dynamically attend to different tokens depending on the features, e.g., using different values for the background and foreground as seen in the spatial cross-attention maps in Fig. 5b.

3.4.2 Different Tokens per Frame. For the spatial cross-attention, the default SVD broadcasts the image embedding across all frames . We argue that we can obtain a higher temporal granularity of the motion by using a different set of tokens for each frame, i.e., 𝐹 × 𝑁 tokens 5 . Having different sets of tokens leads to different sets of keys and values for each frame. Different keys allow the model to attend to different spatial locations for different frames, e.g., attending to the arm of a person in one frame but to the leg in another frame. Different values allow the model to apply different changes to the features for different frames, e.g., applying a vector to inform the model to shift pixels in one direction in one frame but another direction in another frame. This is visualized in Fig. 5b, where the spatial cross-attention maps differ greatly between frames because they use different tokens.

For the temporal cross-attention, the default SVD broadcasts the image embedding across all spatial locations . Inflating this analogously to the spatial case would mean inflating the embedding across the spatial dimensions. Learning different tokens for different

5 Note that we always use the same 𝐹 frames of the motion reference video when optimizing the motion-text embedding.

d

S

T

S

Spatial ResNet block

Temporal ResNet block

Spatial attention block

T

Temporal attention block

0

Image embedding

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00458v1-figure-baa7ac34-f2d2-4484-bfd4-a83831e7bbf9.jpg)

e

(a) Default SVD: Since the image embedding e has only one token, every spatial and temporal location attends 100% to that single token. The cross-attention operation thus degenerates to a simple addition of a single broadcasted vector to the feature tensor.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00458v1-figure-18ea9175-9aea-4450-b897-b07ff429cf78.jpg)

(

)

(

)

H i ( ) W i ( ) F Spatial cross-attention maps Temporal cross-attention maps (b) Inflated SVD (Ours): By introducing more tokens in the token dimension ( 𝑁 ), every spatial and temporal location can dynamically attend to different tokens, e.g., different tokens for the foreground vs. background. For the spatial cross-attention, we use different tokens per frame, resulting in different keys and values per frame. This enables a higher temporal granularity of the motion.

H i

*

W i

Fig. 5. High-level visualization of our motion-text embedding and crossattention inflation. The SVD [Blattmann et al. 2023a] UNet is composed of several levels of blocks, shown in gray, that have similar structure. We visualize the sub-blocks of level 𝑖 and their cross-attention maps in more detail. Our inflated motion-text embedding produces more meaningful crossattention maps, resulting in improved motion learning. The cross-attention maps were extracted from the example of the woman doing jumping jacks in Fig. 4.

spatial locations is nontrivial (spatial dimensions change depending on the input resolution and the level 𝑖 ) and would likely cause issues with the alignment (see Section 3.2.2). Furthermore, in our empirical experiments, the temporal cross-attention seemed to have a smaller effect on the generated motion than the spatial crossattention. Therefore, we decided to keep the number of tokens of our temporal motion-text embedding at 𝑁 but learn them separately

Concatenation

1

•

from the 𝐹 × 𝑁 tokens of the spatial motion-text embedding. Our combined motion-text embedding thus has ( 𝐹 + 1 ) × 𝑁 tokens per motion reference video. Section B.4 describes the tensor operations and dimensions of our motion-text embedding and cross-attention inflation in detail.

To give an intuitive analogy for our motion-text embedding inflation, think of building a house. Instead of using a single tool for every part of the house, it is more efficient to have 𝑁 different tools depending on the spatial location on a given floor - like a hammer for the floor and a drill for the wall. Moreover, each of the 𝐹 floors of the house might need a different set of tools. For example, the roof requires different tools compared to the walls. Similarly, in our approach, we use multiple tokens to handle different aspects of the motion.

## 4 EXPERIMENTS

## 4.1 Implementation Details

Webase our implementation on the 14-frame version of Stable Video Diffusion (SVD) [Blattmann et al. 2023a], but our method could in theory be applied to other image-to-video methods that have a text/image embedding input. Per default, we use 𝑁 = 5 different tokens for each of the 𝐹 = 14 frames, so a total of ( 14 + 1 ) × 5 = 75 tokens for the motion-text embedding. Please refer to Section B for the remaining hyperparameters and implementation details.

Our data consists of motion reference videos and target images from internal data sets as well as target images generated with Stable Diffusion XL [Podell et al. 2023]. We only use one reference video for the optimization per motion and can apply this motion to various target images.

## 4.2 Motion-Text Embedding Analysis

SVD was pre-trained as a text-to-video model and dropped the image (latent) input for some percentage of training iterations for classifier-free guidance [Ho and Salimans 2022]. We find that SVD can produce somewhat reasonable videos with the image (latent) input zeroed out and only the CLIP [Radford et al. 2021] image embedding as input, especially if we increase the classifier-free guidance scale. We can use this to visualize our learned motiontext embedding with an unconditional appearance 6 . Fig. 6 shows motion visualizations of our motion-text embedding for a 'jumping jacks' motion after different numbers of optimization iterations and the generated videos for a given target image side-by-side. Starting around iteration 500, a person doing a 'jumping jacks' motion can be seen in the visualizations. Beyond 1000 iterations, the motion visualizations become more abstract, but the generated motions in the conditional case remain of high quality. Notably, the appearance and position of the people do not match those of the motion reference video (from Fig. 1). Furthermore, the position of the people is different in the conditional and unconditional videos, but all videos have a similar semantic motion. This demonstrates that our motion-text embedding neither encodes the appearance nor the exact spatial positioning of the objects extensively, likely for reasons described in Section 3.2.

6 Note that the visualization is not always easily interpretable, depending on the motion, the optimization iteration, and the seed.

Fig. 6. Motion visualization. We generate videos using our optimized motion-text embedding for a 'jumping jacks' motion (reference from Fig. 1) both with the image input (conditional) and without (unconditional) after a different number of optimization iterations. Note how the appearance of the unconditional generations differs from the motion reference video and varies with different seeds. Further observe that our method effectively generates similar semantic motions without needing or enforcing spatial alignment.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00458v1-figure-a0d13539-9994-4070-8fc4-a30f6a409aee.jpg)

## 4.3 Qualitative Evaluation

As our baseline, we use SVD [Blattmann et al. 2023a] without any adaptations. Since it does not have a motion control input, we do not expect it to follow the correct motion given only the starting image in most cases. However, we show it anyway to showcase the quality and motions of typical SVD outputs. To our knowledge, our method is the first method to address the general motion transfer task in the image-to-video setting. Therefore, there are few methods to choose from that permit a direct and fair comparison. We considered the two most similar classes of methodology and compared our method with a representative of each class. Methods within these classes tend to have certain drawbacks in common:

- · Image-to-video methods with explicit motion representation: Without proper alignment, these methods transfer spatial but not semantic motion and may leak the reference video's structure. Since it is unclear how to automatically extract sparse motion inputs, we compare to VideoComposer [Wang et al. 2024b], which uses dense motion vectors.
- · Text-to-video methods with implicit motion representation: These methods do not directly take a target image input, compromising the preservation of the target's appearance and layout. We compare to MotionDirector [Zhao et al. 2023a] since it learns the target image's appearance, which we expect to perform better than specifying the appearance with text alone.

We use the official implementations. For the models requiring a text input, we use BLIP [Li et al. 2022] to generate image captions and SpaceTimeGPT [Wang 2020] to generate video captions.

Fig. 7 shows motion transfer results for three videos. The SVD baseline produces high-quality videos for the robot and waterfall, but their motions do not match the reference videos. For the face video, SVD introduces artifacts and changes the person's identity with head movement. In our experience, SVD often generates static objects with moving cameras, which users might find unexpected or frustrating. We suggest that motion transfer methods, like ours, can help generate more natural and diverse motions. VideoComposer fails to transfer motions successfully, alters appearances, and sometimes introduces strong artifacts. This issue is inherent to its class and arises from the dense motion inputs, which implicitly encode the reference video's structure that may not align with the input image. Since proper alignment is difficult to achieve in our general (and potentially cross-domain) setting, we find it easier to use an implicit motion representation without a spatial dimension, like ours, for general semantic video motion transfer. MotionDirector correctly transfers motion in the head-nodding example but fails to produce the correct motion for the other two. Furthermore, the appearance does not match the target image exactly. For example, the robot is rotated, the person's identity changes, and the waterfall also looks differently. This limitation stems from text-to-video methods not having a direct image input and relying on learning the appearance (or predicting it from text). Thus, we believe that image-to-video models have inherent advantages over text-to-video models when animating a given image. Our method uniquely preserves the input image's appearance and layout while successfully transferring the semantic motion of the video.

## 4.4 Results

Our motion representation is highly versatile, allowing our method to work with nearly any type of object and motion, as demonstrated in Fig. 1 and Fig. 8. Notably, we do not require a spatial alignment, so our motion-text embedding can be applied to images where objects have different positions and orientations from those in the reference video. For instance, in the sixth row on the right side of Fig. 8, the camera tracks the moving camper van in a similar way to how it tracks the car in the fifth row, despite their misalignment. Furthermore, our method applies the motion to all semantically

Fig. 7. Qualitative evaluation. We compare our method to SVD = Stable Video Diffusion [Blattmann et al. 2023a] (baseline, no motion input), MD = MotionDirector [Zhao et al. 2023a], and VC = VideoComposer [Wang et al. 2024b] for three different motions and target images: full-body reenactment, face reenactment, and camera motion.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00458v1-figure-2977bd8e-7423-4644-93fa-76051f04c9e2.jpg)

Fig. 8. Results. Our method can successfully transfer semantic video motion across a wide number of domains and motions.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00458v1-figure-652c1196-2b3b-43ba-bbe0-b701aece11b6.jpg)

reasonable objects simultaneously 'for free'. It can even handle simple hand-crafted motions, enabling artists to sketch a motion (e.g., drawing stick figures) and apply it to complex scenes. For more results, including joint subject and camera motion as well as extreme cross-domain transfers and applying the same motion to multiple target images, please refer to Section C.

## 4.5 Ablation Study

Our proposed motion-text embedding inflation is crucial for highquality motion transfers. Fig. 9 illustrates different settings for the motion-text embedding size. With only one token, only a small portion of the motion is transferred. Increasing the number of tokens but sharing them across all frames offers some improvement. However, the key factor is having different tokens per frame . Both rows 2 and 3 use 15 tokens, but the version with different tokens per frame performs significantly better. This is logical since the motion-text

Fig. 9. Ablation. Our proposed motion-text embedding inflation is crucial for successful motion transfer. While adding more tokens (increasing 𝑁 ) improves the results already, the biggest gain comes from having different tokens for each frame (where 𝐹 ′ = 𝐹 + 1 = 15 ).

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00458v1-figure-89268f31-656d-4058-9e54-05626eeaba47.jpg)

embedding can adapt to each frame, which is particularly beneficial for complex motions. Increasing the number of tokens per frame further improves performance slightly, but it eventually saturates, so we default to using 𝑁 = 5.

## 4.6 Limitations

Fig. 10 shows typical failure cases of our method. Since we do not fine-tune the model, our method inherits the priors and quality of our pre-trained image-to-video model. We observed that the SVD baseline often struggles with object motions, as can be seen in the head example in Fig. 7, where the appearance changes throughout the video. Our method's results have similar issues: in the first example of Fig. 10, the identity of the target person changes when he moves his head to the side. We believe our motion-text embedding does not exacerbate these issues or temporal inconsistencies, as it primarily instructs the model on the desired motion without altering the rest of the model. Often, it seems that the model attempts to produce the desired motion, but its priors are insufficient to generate a satisfactory result. SVD also does not seem to be able to handle some combinations of motions and given input images, likely because they fall outside of the range of the training data set. When the domain gap between motion reference video and target image is too large, our method may leak the structure of the motion reference video into the generated video. In the second example of Fig. 10, when applying a laid-back walking style to a kangaroo, the kangaroo starts walking, but its feet and overall structure become more human-like. Lastly, we found that some motions are not transferred or to a smaller extent. This is especially visible if a video has multiple motions, where the more fine-grained motion is sometimes not transferred. In the third example of Fig. 10, the person pretends to squat down and type on a keyboard. The dinosaurs in the generated video do squat down, but their hands do not move. We hypothesize that fine-grained motions are also a general limitation of SVD. Overall, we expect better results of our method as image-to-video models improve.

Fig. 10. Failure cases. Our method is limited by the priors and quality of the pre-trained image-to-video model, which may lead to artifacts (e.g., identity changes as head moves in first example). Furthermore, there may be some structure leakage in some cases, leading to certain characteristics from the motion reference video being visible (e.g., human-like legs on a kangaroo in second example). Lastly, our method struggles to transfer spatially finegrained motion at times (e.g., typing motion not transferred to dinosaurs in third example).

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00458v1-figure-1a8ac9fe-fc0e-4d85-9a93-bc43504117ae.jpg)

We believe our approach is more accessible than approaches that require substantial training or fine-tuning. Nevertheless, our approach does require an optimization procedure that takes around one hour on an NVIDIA Tesla A100 (80 GB) GPU per motion. We encourage future work in reducing this per-motion optimization time as well as trying to learn a model to directly predict motion-text embeddings from motion reference videos.

## 5 CONCLUSION

We introduce the general task of transferring the semantic motion of a reference video to any target image. We observe and exploit inherent advantages of image-to-video over text-to-video models for this task and find that text/image embedding tokens are well-suited as a motion representation. Specifically, our method, motion-textual inversion , optimizes an inflated version of the text/image embedding for a given motion reference video. Due to its general nature, this motion can then be applied to a wide number of objects and domains. Our method thus enables completely novel applications and takes a significant step towards being able to reenact anything.

## ACKNOWLEDGEMENTS

We would like to thank Michael Bernasconi, Dominik Borer, Jakob Buhmann, and Daniela Kansy for providing motion videos as well as all participants featured in our internal datasets. We also want to give special thanks to Michael Bernasconi, Vukasin Bozic, Karlis Briedis, Pascal Chang, Guilherme Haetinger, Christopher Otto, Lucas Relic, Seyedmorteza Sadat, and Agon Serifi for their valuable and insightful discussions throughout the project.

## REFERENCES

Sherwin Bahmani, Ivan Skorokhodov, Aliaksandr Siarohin, Willi Menapace, Guocheng Qian, Michael Vasilkovsky, Hsin-Ying Lee, Chaoyang Wang, Jiaxu Zou, Andrea Tagliasacchi, et al. 2024. VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control. arXiv preprint arXiv:2407.12781 (2024).

Jianhong Bai, Tianyu He, Yuchi Wang, Junliang Guo, Haoji Hu, Zuozhu Liu, and Jiang Bian. 2024. UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editing. arXiv preprint arXiv:2402.13185 (2024).

Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et al. 2024. Lumiere: A space-time diffusion model for video generation. arXiv preprint arXiv:2401.12945 (2024).

Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. 2023a. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 (2023).

Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. 2023b. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 22563-22575.

Duygu Ceylan, Chun-Hao P Huang, and Niloy J Mitra. 2023. Pix2video: Video editing using image diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 23206-23217.

- Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros. 2019. Everybody dance now. In Proceedings of the IEEE/CVF international conference on computer vision . 5933-5942.

Changgu Chen, Junwei Shu, Lianggangxu Chen, Gaoqi He, Changbo Wang, and Yang Li. 2024. Motion-Zero: Zero-Shot Moving Object Control Framework for DiffusionBased Video Generation. arXiv preprint arXiv:2401.10150 (2024).

Tsai-Shien Chen, Chieh Hubert Lin, Hung-Yu Tseng, Tsung-Yi Lin, and Ming-Hsuan Yang. 2023a. Motion-conditioned diffusion model for controllable video synthesis. arXiv preprint arXiv:2304.14404 (2023).

Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. 2023b. Control-a-video: Controllable text-to-video generation with diffusion models. arXiv preprint arXiv:2305.13840 (2023).

Zuozhuo Dai, Zhenghao Zhang, Yao Yao, Bingxue Qiu, Siyu Zhu, Long Qin, and Weizhi Wang. 2023. AnimateAnything: Fine-Grained Open Domain Image Animation with Motion Guidance. arXiv e-prints (2023), arXiv-2311.

Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Aleksei Ivakhnenko, Victor Lempitsky, and Egor Zakharov. 2022. Megaportraits: One-shot megapixel neural head avatars. In Proceedings of the 30th ACM International Conference on Multimedia . 2663-2671.

Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. 2022. An image is worth one word: Personalizing text-toimage generation using textual inversion. arXiv preprint arXiv:2208.01618 (2022).

Daniel Garibi, Or Patashnik, Andrey Voynov, Hadar Averbuch-Elor, and Daniel CohenOr. 2024. ReNoise: Real Image Inversion Through Iterative Noising. arXiv preprint arXiv:2403.14602 (2024).

Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. 2023. Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint arXiv:2307.10373 (2023).

Yuchao Gu, Yipin Zhou, Bichen Wu, Licheng Yu, Jia-Wei Liu, Rui Zhao, Jay Zhangjie Wu, David Junhao Zhang, Mike Zheng Shou, and Kevin Tang. 2023. Videoswap: Customized video subject swapping with interactive semantic point correspondence. arXiv preprint arXiv:2312.02087 (2023).

Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang, Pengfei Wan, and Di Zhang. 2024. LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control. arXiv preprint arXiv:2407.03168 (2024).

Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. 2023. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725 (2023).

Sai Sree Harsha, Ambareesh Revanur, Dhwanit Agarwal, and Shradha Agrawal. 2024. GenVideo: One-shot target-image and shape aware video editing using T2I diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 7559-7568.

Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. 2024. CameraCtrl: Enabling Camera Control for Text-to-Video Generation. arXiv preprint arXiv:2404.02101 (2024).

Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. 2022. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 (2022).

- Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising diffusion probabilistic models. Advances in neural information processing systems 33 (2020), 6840-6851.
- Jonathan Ho and Tim Salimans. 2022. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 (2022).

11

Chen Hou, Guoqiang Wei, Yan Zeng, and Zhibo Chen. 2024. Training-free Camera Control for Video Generation. arXiv preprint arXiv:2406.10126 (2024).

- Gee-Sern Hsu, Chun-Hung Tsai, and Hung-Yi Wu. 2022. Dual-generator face reenactment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 642-650.
- Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021).
- Li Hu. 2024. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 8153-8163.
- Teng Hu, Jiangning Zhang, Ran Yi, Yating Wang, Hongrui Huang, Jieyu Weng, Yabiao Wang,andLizhuangMa.2024. MotionMaster: Training-free Camera Motion Transfer For Video Generation. arXiv preprint arXiv:2404.15789 (2024).
- Yash Jain, Anshul Nasery, Vibhav Vineet, and Harkirat Behl. 2023. PEEKABOO: Interactive Video Generation via Masked-Diffusion. arXiv preprint arXiv:2312.07509 (2023).

Hyeonho Jeong, Geon Yeong Park, and Jong Chul Ye. 2023. VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models. arXiv preprint arXiv:2312.00845 (2023).

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. 2022. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems 35 (2022), 26565-26577.

- Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
- Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. https://doi.org/10.48550/ARXIV.2201.12086
- Mingxiao Li, Bo Wan, Marie-Francine Moens, and Tinne Tuytelaars. 2024a. Animate Your Motion: Turning Still Images into Dynamic Videos. arXiv preprint arXiv:2403.10179 (2024).

Weichuang Li, Longhao Zhang, Dong Wang, Bin Zhao, Zhigang Wang, Mulin Chen, Bang Zhang, Zhongjian Wang, Liefeng Bo, and Xuelong Li. 2023. One-shot highfidelity talking-head synthesis with deformable neural radiance field. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 17969-17978.

Yaowei Li, Xintao Wang, Zhaoyang Zhang, Zhouxia Wang, Ziyang Yuan, Liangbin Xie, Yuexian Zou, and Ying Shan. 2024b. Image Conductor: Precision Control for Interactive Video Synthesis. arXiv preprint arXiv:2406.15339 (2024).

Pengyang Ling, Jiazi Bu, Pan Zhang, Xiaoyi Dong, Yuhang Zang, Tong Wu, Huaian Chen, Jiaqi Wang, and Yi Jin. 2024. MotionClone: Training-Free Motion Cloning for Controllable Video Generation. arXiv preprint arXiv:2406.05338 (2024).

Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia. 2023. Video-p2p: Video editing with cross-attention control. arXiv preprint arXiv:2303.04761 (2023).

- Wan-Duo Kurt Ma, JP Lewis, and W Bastiaan Kleijn. 2023. TrailBlazer: Trajectory Control for Diffusion-Based Video Generation. arXiv preprint arXiv:2401.00896 (2023).
- Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Siran Chen, Xiu Li, and Qifeng Chen. 2024. Follow your pose: Pose-guided text-to-video generation using posefree videos. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 38. 4117-4125.

Joanna Materzynska, Josef Sivic, Eli Shechtman, Antonio Torralba, Richard Zhang, and Bryan Russell. 2023. Customizing motion in text-to-video diffusion models. arXiv preprint arXiv:2312.04966 (2023).

Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. 2023. Nulltext inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 6038-6047.

Eyal Molad, Eliahu Horwitz, Dani Valevski, Alex Rav Acha, Yossi Matias, Yael Pritch, Yaniv Leviathan, and Yedid Hoshen. 2023. Dreamix: Video diffusion models are general video editors. arXiv preprint arXiv:2302.01329 (2023).

- Chong Mou, Mingdeng Cao, Xintao Wang, Zhaoyang Zhang, Ying Shan, and Jian Zhang. 2024. ReVideo: Remake a Video with Motion and Content Control. arXiv preprint arXiv:2405.13865 (2024).

Yuval Nirkin, Yosi Keller, and Tal Hassner. 2019. Fsgan: Subject agnostic face swapping and reenactment. In Proceedings of the IEEE/CVF international conference on computer vision . 7184-7193.

Muyao Niu, Xiaodong Cun, Xintao Wang, Yong Zhang, Ying Shan, and Yinqiang Zheng. 2024. MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Model. arXiv preprint arXiv:2405.20222 (2024).

Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and JunYan Zhu. 2023. Zero-shot image-to-image translation. In ACM SIGGRAPH 2023 Conference Proceedings . 1-11.

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. 2023. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 (2023).

Haonan Qiu, Zhaoxi Chen, Zhouxia Wang, Yingqing He, Menghan Xia, and Ziwei Liu. 2024. FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Models. arXiv preprint arXiv:2406.16863 (2024).

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning . PMLR, 8748-8763.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 1, 2 (2022), 3.

Yixuan Ren, Yang Zhou, Jimei Yang, Jing Shi, Difan Liu, Feng Liu, Mingi Kwon, and Abhinav Shrivastava. 2024. Customize-A-Video: One-Shot Motion Customization of Text-to-Video Diffusion Models. arXiv preprint arXiv:2402.14780 (2024).

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 10684-10695.

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. 2023. Dreambooth: Fine tuning text-to-image diffusion models for subjectdriven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 22500-22510.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. 2022. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems 35 (2022), 36479-36494.

Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. 2019. First order motion model for image animation. Advances in neural information processing systems 32 (2019).

Aliaksandr Siarohin, Oliver J Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. 2021. Motion representations for articulated animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 13653-13662.

Jiaming Song, Chenlin Meng, and Stefano Ermon. 2020a. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020).

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. 2020b. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 (2020).

Maham Tanveer, Yizhi Wang, Ruiqi Wang, Nanxuan Zhao, Ali Mahdavi-Amiri, and Hao Zhang. 2024. AnaMoDiff: 2D Analogical Motion Diffusion via Disentangled Denoising. arXiv preprint arXiv:2402.03549 (2024).

Shuyuan Tu, Qi Dai, Zhi-Qi Cheng, Han Hu, Xintong Han, Zuxuan Wu, and Yu-Gang Jiang. 2024. Motioneditor: Editing video motion via content-aware diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 7882-7891.

Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. 2023. Plug-and-play diffusion features for text-driven image-to-image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 1921-1930.

Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. 2022. Diffusers: State-of-the-art diffusion models. https: //github.com/huggingface/diffusers.

Caelen Wang. 2020. SpaceTimeGPT - A Spatiotemporal Video Captioning Model. https://huggingface.co/Neleac/SpaceTimeGPT

Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. 2023b. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06571 (2023).

Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guoqiang Wei, Liping Yuan, and Hang Li. 2024c. Boximator: Generating Rich and Controllable Motions for Video Synthesis. arXiv preprint arXiv:2402.01566 (2024).

Luozhou Wang, Guibao Shen, Yixun Liang, Xin Tao, Pengfei Wan, Di Zhang, Yijun Li, and Yingcong Chen. 2024a. Motion Inversion for Video Customization. arXiv preprint arXiv:2403.20193 (2024).

Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. 2021. One-shot free-view neural talking-head synthesis for video conferencing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 10039-10049.

Wen Wang, Yan Jiang, Kangyang Xie, Zide Liu, Hao Chen, Yue Cao, Xinlong Wang, and Chunhua Shen. 2023a. Zero-shot video editing using off-the-shelf image diffusion models.

Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. 2024b. Videocomposer: Compositional video synthesis with motion controllability. Advances in Neural Information Processing Systems 36 (2024).

Zhouxia Wang, Ziyang Yuan, Xintao Wang, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. 2023c. Motionctrl: A unified and flexible motion controller for video generation. arXiv preprint arXiv:2312.03641 (2023).

Yujie Wei, Shiwei Zhang, Zhiwu Qing, Hangjie Yuan, Zhiheng Liu, Yu Liu, Yingya Zhang, Jingren Zhou, and Hongming Shan. 2023. Dreamvideo: Composing your dream videos with customized subject and motion. arXiv preprint arXiv:2312.04433

## (2023).

- Jianzong Wu, Xiangtai Li, Yanhong Zeng, Jiangning Zhang, Qianyu Zhou, Yining Li, Yunhai Tong, and Kai Chen. 2024b. MotionBooth: Motion-Aware Customized Text-to-Video Generation. arXiv preprint arXiv:2406.17758 (2024).

Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. 2023b. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 7623-7633.

Ruiqi Wu, Liangyu Chen, Tong Yang, Chunle Guo, Chongyi Li, and Xiangyu Zhang. 2023a. Lamp: Learn a motion pattern for few-shot-based video generation. arXiv preprint arXiv:2310.10769 (2023).

Wejia Wu, Zhuang Li, Yuchao Gu, Rui Zhao, Yefei He, David Junhao Zhang, Mike Zheng Shou, Yan Li, Tingting Gao, and Di Zhang. 2024a. DragAnything: Motion Control for Anything using Entity Representation. arXiv preprint arXiv:2403.07420 (2024).

- Dejia Xu, Weili Nie, Chao Liu, Sifei Liu, Jan Kautz, Zhangyang Wang, and Arash Vahdat. 2024. CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation. arXiv preprint arXiv:2406.02509 (2024).
- Wilson Yan, Andrew Brown, Pieter Abbeel, Rohit Girdhar, and Samaneh Azadi. 2023. Motion-conditioned image animation for video editing. arXiv preprint arXiv:2311.18827 (2023).

Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao. 2024. Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motion. arXiv preprint arXiv:2402.03162 (2024).

Shuai Yang, Yifan Zhou, Ziwei Liu, , and Chen Change Loy. 2023. Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation. In ACM SIGGRAPH Asia Conference Proceedings .

Danah Yatim, Rafail Fridman, Omer Bar Tal, Yoni Kasten, and Tali Dekel. 2023. SpaceTime Diffusion Features for Zero-Shot Text-Driven Motion Transfer. arXiv preprint arXiv:2311.17009 (2023).

Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. 2023. Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv preprint arXiv:2308.08089 (2023).

Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. 2023b. I2vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models. arXiv preprint arXiv:2311.04145 (2023).

Yuxin Zhang, Fan Tang, Nisha Huang, Haibin Huang, Chongyang Ma, Weiming Dong, and Changsheng Xu. 2023a. MotionCrafter: One-Shot Motion Customization of Diffusion Models. arXiv preprint arXiv:2312.05288 (2023).

Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. 2023c. Controlvideo: Training-free controllable text-to-video generation. arXiv preprint arXiv:2305.13077 (2023).

Jian Zhao and Hui Zhang. 2022. Thin-plate spline motion model for image animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 3657-3666.

- Rui Zhao, Yuchao Gu, Jay Zhangjie Wu, David Junhao Zhang, Jiawei Liu, Weijia Wu, Jussi Keppo, and Mike Zheng Shou. 2023a. Motiondirector: Motion customization of text-to-video diffusion models. arXiv preprint arXiv:2310.08465 (2023).

Yuyang Zhao, Enze Xie, Lanqing Hong, Zhenguo Li, and Gim Hee Lee. 2023b. Makea-protagonist: Generic video editing with an ensemble of experts. arXiv preprint arXiv:2305.08850 (2023).

- Shenhao Zhu, Junming Leo Chen, Zuozhuo Dai, Yinghui Xu, Xun Cao, Yao Yao, Hao Zhu, and Siyu Zhu. 2024. Champ: Controllable and consistent human image animation with 3d parametric guidance. arXiv preprint arXiv:2403.14781 (2024).
- Yi Zuo, Lingling Li, Licheng Jiao, Fang Liu, Xu Liu, Wenping Ma, Shuyuan Yang, and Yuwei Guo. 2024. Edit-Your-Motion: Space-Time Diffusion Decoupling Learning for Video Motion Editing. arXiv preprint arXiv:2405.04496 (2024).

•

## A BROADER IMPACT AND ETHICS

To the best of our knowledge, our method is the first that can reenact a wide array of objects and motions given a target image and motion reference video without training domain-specific models. We believe this represents a significant advancement in controllable video generation, as our approach can address multiple existing domain-specific scenarios within a single framework and even facilitate entirely new applications. That said, we acknowledge the potential for misuse of reenactment methods like ours, such as creating realistic deepfakes or videos depicting individuals or objects performing specified, potentially inappropriate actions. We strongly condemn such misuse and advocate for implementing safety mechanisms and procedures in real-world applications. Additionally, we support ongoing research into detecting fake videos to mitigate these risks.

## B IMPLEMENTATION DETAILS

## B.1 High-Level Overview of the Implementation

To aid in reproducibility, we list the main steps of our method's implementation below:

- (1) [Only once] Take pre-trained Stable Video Diffusion (SVD) [Blattmann et al. 2023a] and adapt code to inflate motion-text embedding and cross-attention. See high-level description in Section 3.4 and details in Section B.4.
- (2) Initialize motion-text embedding of shape ( 𝐹 + 1 ) 𝑥𝑁𝑥𝑑 . See Section B.2.
- (3) Repeat until convergence:
- · Load same 𝐹 frames of reference video in data loader for each iteration.
- · Augment data. See Section B.2.
- · Input noisy version of frames, motion-text embedding, and other inputs into SVD.
- · Apply loss from Equation 3 to update motion-text embedding.
- (4) Save motion-text embedding.
- (5) For all target images:
- · Input learned motion-text embedding along with new target image to inflated SVD during inference to generate video with motion from reference video.

## B.2 Hyperparameters

Our implementation builds up on the diffusers implementation [von Platen et al. 2022] of Stable Video Diffusion (SVD) [Blattmann et al. 2023a]. We use the default parameters of the 14-frame version of SVD (e.g., micro-conditionings) unless specified otherwise. Like SVD, we generally employ a classifier-free guidance [Ho and Salimans 2022] scale that increases linearly from 1 to 3 across the frame axis. For the motion visualization (unconditional image input), however, we use a higher scale, i.e., increasing linearly from 1 to 10, to improve the visibility of the objects. We initialize the 𝐹 = 14 sets of 𝑁 = 5 tokens for the spatial cross-attention with the CLIP image embedding token of each corresponding frame and the 𝑁 = 5 tokens for the temporal cross-attention with the mean of the CLIP image embedding tokens across all frames. We additionally add Gaussian noise N( 0 0 1 , . ) to the combined motion-text embedding during initialization. In our experience, the initialization does not affect the results significantly, so other initializations are equally reasonable. During optimization, we always pick the same 𝐹 frames of a given video and apply the same spatial and color augmentations to all frames 7 . Since most of the video motion is determined in noisy diffusion steps, we shift the noise schedule towards higher noise values (from 𝑃 mean = 1 0 . , 𝑃 std = 1 6 to . 𝑃 mean = 2 8 . , 𝑃 std = 1 6 where log . 𝜎 ∼ N( 𝑃 mean , 𝑃 2 std ) to speed up the optimization. We use Adam [Kingma and Ba 2014] with a learning rate of 1 𝑒 -2 for 1000 iterations with a batch size of 1.

## B.3 Hardware Requirements and Runtime

The optimization for a motion reference video with a resolution of 1024 × 576 takes around 45 GB of GPU memory and around one hour on an NVIDIA Tesla A100 (80 GB) GPU. The inference takes less than one minute per video. Note that our implementation is not optimized extensively for memory or runtime.

## B.4 Motion-Text Embedding and Cross-Attention Inflation

This section provides more implementation details for the motion-text embedding and cross-attention inflation described in Section 3.4. Fig. 11 shows the spatial and temporal cross-attention layers of the default Stable Video Diffusion (SVD) [Blattmann et al. 2023a] and our inflated version along with their tensor dimensions.

The image embedding of the default SVD consists of a single token and has dimensions 𝐵 × 1 × 𝑑 , where 𝐵 is the batch size (in our implementation typically 1 when optimizing the motion-text embedding and 2 during inference because of classifier-free guidance) and 𝑑 is the CLIP [Radford et al. 2021] embedding dimension. For spatial cross-attention, the image embedding is broadcast to dimensions ( 𝐵 ∗ 𝐹 ) × 1 × 𝑑 , i.e., the same token is used for all 𝐹 frames. This results in an attention map 𝑀 of dimensions ( 𝐵 ∗ 𝐹 ) × ( 𝐻 𝑖 ∗ 𝑊 𝑖 ) × 1 where 𝐻 𝑖 and 𝑊 𝑖 are the spatial heights and widths respectively, and 𝐶 𝑖 is the number of channels of level 𝑖 of the diffusion model. Notably, due to the

7 For horizontal camera motions, we turn of horizontal flipping

e

ing

e

s in

(

B

(

*

F

B

*

H

) ×

C

*

W

: (

B

*

H

eeded

B

poral

×

) ×

H

×

F

×

W

C

*

W

) ×

F

×

QK T

QK T

d a

: (

(

B

WK

K

*

H

:

(

B

B

*

H

*

W

) ×

softmax

*

W

*

B

) × 1 ×

H

*

F

(

× 1

QK T

d

a

(

B

FC

(

R

(

B

*

F

Features out

(a) Default SVD [Blattmann et al. 2023a]: Since the image embedding e has only one token, the softmax operation causes all entries of the cross-attention maps to be 1 . Therefore, the section highlighted in yellow simplifies to a broadcasted version of the value vector of that token.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00458v1-figure-b5501f76-f063-49c3-bc3b-65eeb8c2943e.jpg)

(b) Inflated SVD [Blattmann et al. 2023a] (Ours): We use 𝑁 tokens instead of 1 , so the model now dynamically attends to different tokens depending on the spatial and temporal location. Additionally, we use different sets of tokens per frame for the spatial cross-attention instead of broadcasting the same tokens to all frames.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00458v1-figure-42008a06-75ea-4364-bd82-423a3ea58cd4.jpg)

Fig. 11. Technical diagrams of the motion-text embedding and cross-attention inflation showing the dimensions of the features of the spatial and temporal cross-attention blocks. The changes between the default SVD [Blattmann et al. 2023a] and our inflated version are shown in red font. 𝐵 = batch size, 𝐹 = number of frames, 𝐶 = number of channels, 𝐻 = height, 𝑊 = width, 𝑑 = embedding dimension, 𝑑 𝑎 = attention dimension, 𝑁 = token dimension, 𝑊 𝑄 = query weight matrix, 𝑊 𝐾 = key weight matrix, 𝑊 𝑉 = value weight matrix, 𝑄 = queries, 𝐾 = keys, 𝑉 = values, FC = fully connected layer. For simplicity, the multiple attention heads and block level 𝑖 indices are not shown.

B

*

)

H

*

H

d

W

) × 1 ×

V

*

(

B

WV

d a

W

) ×

F

*

W

) × (

) ×

H

*

V

×

F

W

d a

×

C

) ×

C

*

H

: (

B

*

*

W

H

) × 1 ×

*

d

W

) × 1 ×

d a

Motion-text embedding

m

Features in

(

B

*

F

R

(

B

WQ

Q

= (

B

*

F

Spatial

) ×

C

) × (

×

H

H

*

W

+1

B

× (

×

W

) ×

C

) × (

H

*

W

) ×

QK T

QK T

= (

*

F

F

F

d a

+1) ×

N

N

×

(

d

B

*

WK

K

= (

B

*

F

) × (

F

d

) ×

B

H

N

*

R

×

F

) ×

*

softmax

(

W

) ×

=

B

×

d

N

N

QK T

d

a

(

)

B

FC

(

B

*

F

Features out

*

F

V

) × (

) × (

×

H

H

F

d a

*

W

*

W

×

F

N

×

d

(

B

WV

V

) ×

) ×

d a

C

*

F

= (

) ×

B

N

N

*

F

×

) ×

•

Fig. 12. Motion style transfer. Our learned motion-text embeddings do not only store the rough motion category but also the style of the motion. Here, we apply two different gaits to the same target image: a horse trot (smooth) and a canter (rocking). The resulting videos for the cartoon dog are not only showing the dog moving, but their motions also closely match the motion reference video's gait style. Furthermore, the extreme cross-domain examples with the boat, car, and cereal box show that the essence of the motion style is preserved even across completely different objects.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00458v1-figure-7fc948da-d762-494c-835c-05b4317a3ffd.jpg)

softmax operation and the last dimension being 1, every value of the attention map is 1. This means that each spatial location attends 100% to the single token. Similarly, for temporal cross-attention, the image embedding is broadcast from dimensions of 𝐵 × 1 × 𝑑 to dimensions ( 𝐵 ∗ 𝐻 𝑖 ∗ 𝑊 𝑖 ) × 1 × 𝑑 , eventually leading to an attention map 𝑀 of dimensions ( 𝐵 ∗ 𝐻 𝑖 ∗ 𝑊 𝑖 ) × 𝐹 × 1 where every value is 1. Having only one token thus leads to a degenerate case of the cross-attention where Attention ( 𝑄, 𝐾,𝑉 ) = 𝑉 (broadcasted) and many of the components (e.g., queries and keys) have no effect on the result.

- B.4.1 Multiple Tokens. To avoid the above degenerate case and instead be able to dynamically attend to different tokens, we extend the token dimension from 1 to 𝑁 where 𝑁 is a hyperparameter. For spatial cross-attention, this results in an attention map 𝑀 of dimensions ( 𝐵 ∗ 𝐹 ) × ( 𝐻 𝑖 ∗ 𝑊 𝑖 ) × 𝑁 where, in general, each spatial location has different values ≠ 1 for the 𝑁 different tokens. Similarly, the temporal cross-attention map 𝑀 has dimensions ( 𝐵 ∗ 𝐻 𝑖 ∗ 𝑊 𝑖 ) × 𝐹 × 𝑁 with values ≠ 1. Since SVD was pre-trained using multiple text embedding tokens as input, the code can already handle multiple tokens, so mainly the initialization of the motion-text embedding as well as some input dimensions have to be adapted slightly.
- B.4.2 Different Tokens per Frame. As explained in Section 3.4, we propose to learn different sets of tokens per frame for the spatial crossattention to obtain a higher temporal granularity of the motion. The default SVD implementation broadcasts the embedding from dimensions 𝐵 × 𝑁 × 𝑑 across all frames to ( 𝐵 ∗ 𝐹 ) × 𝑁 × 𝑑 (where 𝑁 = 1 originally). We instead learn a larger spatial motion-text embedding of dimensions 𝐵 × 𝐹 × 𝑁 × 𝑑 and reshape it to ( 𝐵 ∗ 𝐹 ) × 𝑁 × 𝑑 . We keep the dimensions of the temporal motion-text embedding at 𝐵 × 𝑁 × 𝑑 and learn it separately. Therefore, the dimensions of the combined spatial and temporal motion-text embedding is 𝐵 × ( 𝐹 + 1 ) × 𝑁 × 𝑑 .

## C ADDITIONAL RESULTS

Fig. 12 shows that our method does not only apply the rough motion category but also its style, even in cases where the domains differ vastly, e.g., transferring the motion of a horse to a cereal box. Furthermore, these examples demonstrate that our method can transfer joint subject and camera motion. Fig. 13 shows that our method generates the same semantic rather than spatial motion by applying the same learned motion to a flipped target image. Fig. 14 shows additional results of our method, where we apply the same motion to different target images.

Fig. 13. Semantic motion transfer. Our learned motion-text embeddings store the semantic motion (animal moving in the direction it is facing and moving its head down) rather than the spatial motion (animal moving from right to left and left part is going down). This can be seen in the above example where we apply the same learned motion-text embedding to a flipped input image, and our method produces semantically similar results.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00458v1-figure-c402cc90-0c31-41cb-a3fd-00434cc2b993.jpg)

Fig. 14. Additional results. Our learned motion-text embeddings can be applied to multiple target images, resulting in semantically similar motions.

![Image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00458v1-figure-0c07b80a-47d5-4801-a362-6da18e8dec36.jpg)
Manuel Kansy Jacek Naruniec Christopher Schroers Markus Gross Romann M. Weber
Reenact Anything: Semantic Video Motion Transfer Using Motion-Textual Inversion
Recent years have seen a tremendous improvement in the quality of video
generation and editing approaches. While several techniques focus on editing
appearance, few address motion. Current approaches using text, trajectories, or
bounding boxes are limited to simple motions, so we specify motions with a
single motion reference video instead. We further propose to use a pre-trained
image-to-video model rather than a text-to-video model. This approach allows us
to preserve the exact appearance and position of a target object or scene and
helps disentangle appearance from motion. Our method, called motion-textual
inversion, leverages our observation that image-to-video models extract
appearance mainly from the (latent) image input, while the text/image embedding
injected via cross-attention predominantly controls motion. We thus represent
motion using text/image embedding tokens. By operating on an inflated
motion-text embedding containing multiple text/image embedding tokens per
frame, we achieve a high temporal motion granularity. Once optimized on the
motion reference video, this embedding can be applied to various target images
to generate videos with semantically similar motions. Our approach does not
require spatial alignment between the motion reference video and target image,
generalizes across various domains, and can be applied to various tasks such as
full-body and face reenactment, as well as controlling the motion of inanimate
objects and the camera. We empirically demonstrate the effectiveness of our
method in the semantic video motion transfer task, significantly outperforming
existing methods in this context.