## MotionFix: Text-Driven 3D Human Motion Editing

NIKOS ATHANASIOU, Max Planck Institute for Intelligent Systems, Germany

ALPÁR CSEKE ∗ , Max Planck Institute for Intelligent Systems, Germany and Meshcapade, Germany MARKOS DIOMATARIS, Max Planck Institute for Intelligent Systems, Germany and ETH Zürich, Switzerland MICHAEL J. BLACK, Max Planck Institute for Intelligent Systems, Germany GÜL VAROL, LIGM, École des Ponts, Univ Gustave Eiffel, CNRS, France

Fig. 1. Our text-driven motion diffusion model (TMED) enables 3D human motion editing from natural language descriptions. To train this model, we introduce a semi-automatically collected dataset MotionFix that contains diverse types of editing such as modifying body parts, changing certain moments of the motion, or editing the speed or the style.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-01c81b6d-d396-495b-922d-170ece8b4502.jpg)

The focus of this paper is 3D motion editing . Given a 3D human motion and a textual description of the desired modification, our goal is to generate an edited motion as described by the text. The challenges include the lack of training data and the design of a model that faithfully edits the source motion. In this paper, we address both these challenges. We build a methodology to semi-automatically collect a dataset of triplets in the form of (i) a source motion, (ii) a target motion, and (iii) an edit text, and create the new MotionFix dataset. Having access to such data allows us to train a conditional diffusion model, TMED, that takes both the source motion and the edit text as input. We further build various baselines trained only on text-motion pairs datasets, and show superior performance of our model trained on triplets. We introduce new retrieval-based metrics for motion editing, and establish a new benchmark on the evaluation set of MotionFix. Our results are encouraging, paving the way for further research on finegrained motion generation. Code, models and data are available at our project website.

∗ Work was done while at MPI.

Authors' addresses: Nikos Athanasiou, nathanasiou@tuebingen.mpg.de, Max Planck Institute for Intelligent Systems, Germany; Alpár Cseke, acseke@tuebingen.mpg.de, Max Planck Institute for Intelligent Systems, Germany and Meshcapade, Germany; Markos Diomataris, mdiomataris@tuebingen.mpg.de, Max Planck Institute for Intelligent Systems, Germany and and ETH Zürich, Switzerland; Michael J. Black, black@tuebingen.mpg.de, Max Planck Institute for Intelligent Systems, Germany; Gül Varol, gul.varol@enpc.fr, LIGM, École des Ponts, Univ Gustave Eiffel, CNRS, France.

CCS Concepts: · Computing methodologies → Motion processing .

Additional Key Words and Phrases: Motion Editing, Motion from Instructions

## 1 INTRODUCTION

Human motion control is an essential component of the animation pipeline, and involves creating a motion, as well as editing it until the motion matches the desired outcome. Text descriptions have emerged as one of the prominent ways to control motion generation [Guo et al. 2022a; Petrovich et al. 2022; Tevet et al. 2023; Zhang et al. 2023a]. However, due to the inherent ambiguity in high-level language instructions, the resulting generation may not necessarily correspond to the motion one has in mind. An animator may then need to edit the motion further. Motion editing is non-trivial, arguably more complex than static pose editing, and may involve multiple types of instructions, such as changing the speed of a motion, modifying the repetitions for cyclic actions, adjusting the posture of a particular body or modifying a certain temporal segment of a motion. In this work, we aim, given an initial source motion and an edit description, to generate a new motion that follows the source motion and edits it according to the text instruction.

There are existing approaches that can modify body coordinates [Karunratanakul et al. 2023] and lower/upper limbs [Tevet et al. 2023; Zhang et al. 2023a]. However, they require manual selection of the body parts, which prevents making edits beyond local modifications such as the speed of the overall action. On the other hand, some methods have been proposed to add more fine-grained control to text-to-motion generation; examples include temporal [Athanasiou et al. 2022; Zhang et al. 2023c] and spatial [Athanasiou et al. 2023; Goel et al. 2024; Zou et al. 2024] compositions. These methods can be repurposed for editing, but would be limited to addition or subtraction of actions. Consequently, all these works are limited to specific types of edits. Instead, our work considers unrestricted edits described by language instructions. To this end, we collect a manually annotated dataset, MotionFix, that supports training generative models for the task of text-driven motion editing.

Constructing a dataset for 3D human motion editing is non-trivial. In contrast to text-based image editing, where methods such as InstructPix2Pix [Brooks et al. 2023] exploit large text-to-image generation models [Rombach et al. 2022] to automatically create training data, there exists no 3D motion generation model which generalizes faithfully to unrestricted text inputs. Besides, dynamic edits can be more complex than static image edits. In fact, PoseFix [Delmas et al. 2023] is a successful example of a 3D human body pose editing dataset; however, the difference between static poses are mapped to text in a rule-based manner using joint distances, and this would not be applicable to dynamic motions. In this work, we take a different route and mine existing motion capture (MoCap) datasets to find suitable motion pairs automatically, for which the differences are then manually described by typing text. By not relying on a generative model, we ensure motion quality; and by annotating the text (which is relatively fast), we achieve unrestricted edits.

The key challenge in our semi-automatic data curation pipeline is how to find motion pairs that are similar enough for a meaningful and concise edit text to describe the difference. The difference between the source and target motions should not be too large to avoid the annotator typing a too-complex text or describing entirely the target motion, discarding the source. The motions should have similarities for the annotator to potentially make reference to the source. Our solution is to employ the recent TMR motion embedding space [Petrovich et al. 2023] that effectively captures semantics, as well as sufficient details for the body dynamics, thanks to its contrastively and generatively trained motion encoder. To form our candidate pairs for annotation, for each motion in a large MoCap collection [Mahmood et al. 2019], we retrieve the top-ranked motions according to their embedding similarity. Using crowdsourcing, we collect textual annotations for these pairs. The resulting dataset, MotionFix, is the first text-based motion editing dataset, which contains different types of edits as can be seen in Figure 1 (top). Some edits involve a specific body part (e.g., the hand should 'make the circular motion a bit wider'), others alter the overall body dynamics (e.g., 'make the circle bigger and walk faster' when walking in a circle).

MotionFix enables both training and benchmarking for this new task. We design and train a T ext-based M otion E diting D iffusion model, TMED, that is conditioned on both the source motion and the edit text. The results of our TMED model are encouraging as shown in Figure 1 (bottom), generating different types of edits. For example, the model can edit the overall spatial coordinates of a motion ('do a left side tilt instead of back one'), the way a motion is performed ('sit straight, don't get up'), parts of the body ('do it with the hand raised lower'), or the speed of a motion ('faster').

To benchmark our model and compare against baselines, we introduce new metrics on the evaluation set of MotionFix. Following the commonly adopted retrieval-based metrics in text-to-motion generation benchmarks [Guo et al. 2022a], we perform motionto-motion retrieval and check how often the ground-truth target motion is in the top ranks. We also report the ranking of the source motion to evaluate the proximity to the source. While this metric should not be too high - otherwise there would be no edit - it gives an intuition on whether the generated motion deviates too much from the source. Our experiments demonstrate that our conditional model trained on triplets generates motions that are closer to the target, compared to strong baselines we build on top of state-of-theart text-to-motion generation methods, which have only access to text-motion pairs for training.

Our contributions are the following: (i) We introduce MotionFix, the first language-based motion editing dataset, that provides motion-motion-text triplets annotated through our semi-automatic data collection methodology. This dataset allows both for training and benchmarking for this new task. (ii) We introduce several baselines based on text-to-motion generation, together with editrelevant body parts detection using language models. While our baselines achieve promising results, we show that models trained on text-motion pairs fall behind those trained on our triplets. (iii) We propose TMED, a diffusion-based model for motion editing given language instructions. We demonstrate both qualitatively and quantitatively that TMED outperforms all the baselines.

## 2 RELATED WORK

In the following, we briefly overview relevant works on motion generation, editing, and datasets.

3D motion generation from text. In contrast to the relatively mature areas of text-to-image generation [Rombach et al. 2022] and text-based image editing [Brooks et al. 2023], language-based 3D human motion generation is at its infancy. Initial work employs VAEs [Kingma and Welling 2014] with action label conditioning using a small set of categories [Guo et al. 2020; Petrovich et al. 2021]. With the introduction of recent text-motion datasets [Guo et al. 2022a; Lin et al. 2023; Punnakkal et al. 2021], there has been increased interest in conditioning the generation on free-form language inputs [Guo et al. 2024, 2022a,b; Petrovich et al. 2022; Tevet et al. 2022; Uchida et al. 2024; Zou et al. 2024]. Recently, diffusion models [Ho et al. 2020] have been successfully integrated [Shafir et al. 2024; Tevet et al. 2023; Wan et al. 2023; Xie et al. 2024; Zhang et al. 2023a], producing state-of-the-art results in text-to-motion generation.

Several works focus on increased controllability in motion generation, going beyond a single textual input. Examples include enabling temporal compositionality (a series of motions) [Athanasiou et al. 2022; Lee et al. 2022; Shafir et al. 2024], spatial compositionality [Athanasiou et al. 2023; Zhang et al. 2023a] (simultaneous motions), and a unified framework of timeline control [Petrovich

et al. 2024]. Diffusion-based models were shown to be suitable for fine-grained local control such as joint trajectories [Karunratanakul et al. 2023; Shafir et al. 2024; Xie et al. 2024] or keyframes [Cohan et al. 2024]. Our work is in similar spirit in terms of providing more control to users; however, in contrast to the above, our focus is motion editing using language.

Language-based human body editing. There are numerous traditional methods for editing 3D humans to generate movies [Catmull 1972], imposing space or time constraints [Cohen 1992], interactively generating motions using procedural animation [Lee and Shin 1999; Perlin 1995] or physics-based approaches [Popović and Witkin 1999]. Recent work using language can be grouped into pose [Delmas et al. 2023; Kim et al. 2021] or motion [Fieraru et al. 2021; Goel et al. 2024] editing. In FixMyPose [Kim et al. 2021], the focus is on editing athletic human poses in synthetic images. In [Delmas et al. 2023], a text-based 3D human pose editing method is developed, enabled through the collection of the PoseFix dataset containing language descriptions of differences between pairs of poses. PoseFix builds on the previous work of PoseScript [Delmas et al. 2022], where a dataset of pose descriptions are automatically collected through a rule-based approach. Unlike PoseFix that concentrates on static poses, our MotionFix dataset involves dynamic motions where the space of possible edits are much larger, necessitating a different approach to data collection.

In terms of dynamic bodies, current motion editing approaches can be separated into three categories: (a) Style -based editing or motion style transfer exploits datasets that contain a small set of style labels such as 'angry' and 'old' [Aberman et al. 2020; Kobayashi et al. 2023; Mason et al. 2022]. This line of work, focuses mostly on copying the style of one motion onto another, typically performing the same action. (b) Part -based editing considers selecting a subset of the body. MDM [Tevet et al. 2023] shows the potential of diffusion models to edit the upper/lower body by text-conditioned motion inpainting. Similarly, MotionDiffuse [Zhang et al. 2023a] and FLAME [Kim et al. 2023] manually specify body parts to edit them with text. More recently, CoMo [Huang et al. 2024] and FineMoGen [Zhang et al. 2023b] use LLMs to produce edit texts and demonstrate promising results for part editing. (c) Among heuristic -based approaches [Fieraru et al. 2021; Goel et al. 2024], AIFit [Fieraru et al. 2021] can edit domain-specific exercise poses from a pre-defined grammar and is focused on a limited set of cyclic motions from their Fit3D dataset. Iterative Motion Editing [Goel et al. 2024] relies on captioned source motions, which are passed through an LLM along with a pre-defined set of 'Motion Editing Operators' (MEOs), to detect which joints and frames should be edited. A pre-trained diffusion model is then used to infill these locations. In contrast with the prior work, we do not focus on a specific type of motion editing or any heuristics. Our MotionFix contains diverse edits as can be seen in Figure 1 (top) and Figure 2. Closest to ours is the concurrent work of [Goel et al. 2024]; however, as mentioned above, their approach is not fully automatic due to requiring a captioned source motion. Their keyframe selection heuristic further limits the applicability to certain edit types. Moreover, due to the unavailability of an open-source code at the time of writing this paper, we do not provide comparisons in this work.

Table 1. Comparisonwithexistingdatasets: MotionFix is the first dataset supporting the task of text-based motion editing.

| Dataset                       | #motions   |   vocab. | label type                 |
|-------------------------------|------------|----------|----------------------------|
| KIT-ML [Plappert et al. 2016] | 3911       |     1623 | motion description         |
| BABEL [Punnakkal et al. 2021] | 10881      |     1347 | motion description, action |
| HumanML3D [Guo et al. 2022a]  | 14616      |     5371 | motion description         |
| PoseFix [Delmas et al. 2023]  | 6157 x 2   |     1068 | pose editing               |
| MotionFix (ours)              | 6730 x 2   |     1479 | motion editing             |

3D human &amp; language datasets. The progress in controlling 3D humans with text has been driven by new datasets that pair 3D humans with language descriptions. In Table 1, we summarize the three most popular motion description datasets. KIT [Plappert et al. 2016] is the first source of such data containing textual annotations for ∼ 4k motion sequences. To obtain sufficient data to train deep networks, AMASS [Mahmood et al. 2019] unifies several MoCap collections in the SMPL body format [Loper et al. 2015]; however, it lacks language descriptions. This is addressed by BABEL [Punnakkal et al. 2021] and HumanML3D [Guo et al. 2022a], which concurrently collect semantic annotations in the form of action categories and/or textual descriptions. While KIT, BABEL, HumanML3D enable textto-motion generation training, they do not support editing. On the other hand, as can be next seen in Table 1, PoseFix [Delmas et al. 2023] provides pose editing triplets, but does not support motion editing. Our MotionFix dataset supports motion editing training, while being at a similar scale to PoseFix in terms of the number of triplets and the vocabulary of edit texts.

## 3 THE NEW MOTIONFIX DATASET

Appropriate training data for text-based motion editing would be in the form of triplets: source motions, target motions and edit texts. As discussed in Section 1, a big challenge in motion editing from language instructions is the lack of training data. To overcome this challenge, we design a semi-automatic data creation methodology. Wefirst automatically construct candidate motion pairs that are similar (and different) enough, so the edit can potentially be described by language in simple words. We then ask annotators to manually type the edit text. In the following, we detail our procedure.

We make use of a motion embedding space to find motion pairs that are similar. Specifically, we employ a recent text-to-motion retrieval model TMR [Petrovich et al. 2023]. TMR is trained with a contrastive loss on the latent space of motions and texts, and reports state-of-the-art results for text-motion retrieval. We observe that such a model, by design, has the ability to produce latent motion representations that, for a given motion, ranks the semantically close ones nearby in the embedding space. We then use TMR to perform motion-to-motion retrieval. This is in similar spirit to using CLIP [Radford et al. 2021] for image similarity. We construct our dataset by finding such motion pairs from the AMASS MoCap collection [Mahmood et al. 2019].

From each motion in AMASS, we first extract TMR motion embeddings with sliding windows of 3 to 5 seconds. In preliminary analysis, we found that using longer motion pairs reduces the probability of finding good candidates that differ by simple edits, while using shorter ones usually yields motion pairs that have a high

Fig. 2. Dataset samples: We display source motions (red) overlaid with target motions (green) from our MotionFix dataset, together with their corresponding text annotations.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-b0d4f538-0108-49ed-a310-6baa83070f2f.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-c43a4e97-48b0-4c80-ba3b-6ced14c5c466.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-204b5e24-09db-4f72-91d2-1769c9a34fee.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-09f7f32d-5f17-48c2-b216-ef996b11815a.jpg)

probability to be almost identical. Then, we compute the pairwise embedding similarities and filter out all the motions pairs that have similarity ≥ 0 99 to avoid identical motions. We extract the top-2 . most similar motions for a given motion and include these pairs in the annotation pool. We experimented with thresholding instead of following a top-k selection approach, but the TMR feature similarity is not well calibrated across motion pairs, which would make finding a constant threshold difficult. Finally, we align each motion pair to have the same initial translation and global orientation for the gravity axis to avoid labeling redundant edits that can be trivially created by changing the initial body translation and orientation.

such as spatial edits 'throw from higher', temporal subtraction of actions 'start standing not bent down', mixture of both 'bend down a bit more, stand up faster', and repetitions with adjustments of the whole body motion 'do one more repetition and extend arms and legs wider apart'. We include several visual examples in Figure 2 that show body part editing ('keep arms at shoulder height'), directional ('move in the other direction') and temporal ('start crawling earlier') changes. We provide dynamic video examples in our webpage ∗ through the supplementary video and the data exploration interface. Detailed statistics regarding the texts and motions can also be viewed in Section A of the appendix.

Once we curate a list of candidate motion pairs, we give them to annotators from Amazon Mechanical Turk (AMT). In the annotation interface, along with the instructions, we give representative examples with multiple plausible edit texts for each motion pair. We allow the option to skip motion pairs if they are too similar (no difference to describe), or if they are too different (no easy way to describe the difference). Quantitatively, 7% and 55% of the pairs were considered too similar or too different, respectively. For the remaining pairs, we found that the majority are suitable candidates for editing. Our annotation interface can be visualized in the supplementary materials (sup. mat.).

We performed several rounds of data collection. After the initial round, we observed that some annotators tend to overanalyze the edit, which tends to describe the target motion alone or produce overcomplicated edits. Hence, after computing the statistics for a manually curated set of good annotations, we started encouraging the annotators to keep their edit texts around 3 -12 words, but no longer than 15 words. We explicitly request the annotators to refer to the source motion and encourage them to use words indicative of edit texts, e.g., 'instead', 'higher/lower', 'same/opposite'.

The resulting MotionFix dataset contains 6730 triplets of sourcetarget motions and text annotations. We partition the data into train/validation/test splits randomly with 80%/5%/15% ratios, and obtain 5387/330/1013 triplets for each split, respectively. As shown in Table 1, in contrast to previous motion description datasets that provide text-motion pairs [Guo et al. 2022a; Plappert et al. 2016; Punnakkal et al. 2021], our dataset enables training for motion editing, by also including a source motion. MotionFix is similar in spirit to PoseFix [Delmas et al. 2023], but our labels describe the difference between dynamic motions, as opposed to static poses. Our dataset involves unrestricted edits, leading to different edit types

## 4 TEXT-DRIVEN MOTION EDITING DIFFUSION MODEL

We introduce TMED, a text-driven motion editing diffusion model. Given a short 3D human motion, a textual instruction describing a modification, and a noise vector to enable randomness, the model generates an edited motion. Similar tasks have been addressed in the image domain for text-based image editing [Brooks et al. 2023], from which we take inspiration for our model design. We further build on motion diffusion model (MDM) [Tevet et al. 2023] that takes only text as input and generates a motion. In contrast, our model has an additional condition on the source, thus requires a different training dataset (as described in Section 3). In the following, we present the components of our TMED model.

## 4.1 3D Human Motion Representation

We use a sequence of SMPL [Loper et al. 2015] body parameters to represent a human motion. SMPL is a linear function that maps the shape, and pose parameters of 𝐽 joints, along with the global body translation and orientation, to a 3D mesh. The joint positions, 𝐽 𝑝 , can be obtained from vertices via the learned SMPL joint regressor. Following previous work [Petrovich et al. 2022] that discards the shape parameters, we set the shape parameters to zero (mean shape), since motion is parameterized primarily by pose parameters.

Various alternative representations have been used based on joint positions with respect to the local coordinate system of the body [Guo et al. 2022a; Holden et al. 2016; Starke et al. 2019]. Unlike prior works [Guo et al. 2022a; Tevet et al. 2023; Zhang et al. 2023a] that fit SMPL bodies to skeleton generations, we aim to enable direct regression of SMPL parameters, bypassing the need of a costly

∗ https://motionfix.is.tue.mpg.de

Fig. 3. Models overview: (left) We illustrate our TMED model during training. We noise the target motion for 𝑡 steps, and the transformer model is trained to denoise it back by one step. The conditions - text and source motion - are appended to the input. CLIP backbone is frozen, while components denoted in pink are learned during training. At test time, the iterative diffusion process is initialized from random noise instead of the noised target. (right) Our MDM-BP baseline is repurposed from a pretrained text-to-motion generation model to be used only at test time for motion editing. The model is initialized from random noise and the body parts not to be edited according to GPT are copied from the source motion.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-74347cfd-291a-4d42-b3a0-35452a610833.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-bb5bafeb-8c4c-4904-8b8c-b728993d48d9.jpg)

post-processing optimization [Bogo et al. 2016], and thus making our method ready to use for animation frameworks.

A common approach for representing SMPL pose parameters within a learning framework is to employ 6D rotations [Zhou et al. 2019], and to apply first-frame canonicalization for motions [Athanasiou et al. 2022; Petrovich et al. 2022]. Similarly, we canonicalize our motions prior to training, so that all face the same direction in the first frame and have the same initial global position. Inspired by [Holden et al. 2016; Petrovich et al. 2024], we represent the global body translation as differences between consecutive frames. Supervising with such relative translations helps the denoiser to generate better trajectories, as we observed unsmooth generations when using the absolute translation. Similar to STMC [Petrovich et al. 2024], we factor out the 𝑧 -rotation from the pelvis orientation and separately represent the global orientation as the 𝑥𝑦 -orientation and the 𝑧 -orientation as the differences between rotations in consecutive frames (resulting in 12 features, i.e., 6D representation for 𝑥𝑦 and 𝑧 ). We represent the body pose with 6D rotations [Zhou et al. 2019]. Similar to [Petrovich et al. 2022], we exclude the hand joints as they mostly do not move in the datasets we use. We additionally append the local joint positions after removing 𝑧 -rotation of the body [Holden et al. 2016; Petrovich et al. 2024] (resulting in 192 dimensional features with 6 × 21 for rotations and 22 × 3 for joints including the root joint). Thus, each motion frame has a dimension 𝑑 𝑝 = 207, consisting of 3 features for the global translation, 12 for the global orientation, and 192 for the body pose. The motion is represented as a sequence of the pose representations. During training, all features are normalized according to their mean and variance over the training set.

## 4.2 Conditional Diffusion Model

To learn TMED, we use our new training data, where each data sample comprises a source motion 𝑀 𝑆 , target motion 𝑀 𝑇 , and a language instruction 𝐿 . We train a conditional diffusion model that learns to edit the source motion with respect to the instruction. We design a model similar to that of InstructPix2Pix [Brooks et al. 2023], where the generation from a random noise vector is conditioned on two further inputs 𝐿 and 𝑀 𝑆 . Here, instead of a sequence of image patch tokens, the motion modality is represented as a variablelength sequence of motion frames. The noised target motion, the text condition 𝐿 , and the source motion condition 𝑀 𝑆 are all fed as input to the denoiser at every diffusion step.

Diffusion models [Sohl-Dickstein et al. 2015] learn to gradually turn random noise into a sample from a data distribution by a sequence of denoising autoencoders. This is achieved by a diffusion process that adds noise 𝜖 𝑡 to an input signal, 𝑀 𝑇 . We denote the noise level added to the input signal by using 𝑡 , the diffusion timestep, as a superscript. This produces a diffused sample, 𝑀 𝑡 𝑇 . The amount of noise added at timestep 𝑡 = 1 , . . . , 𝑁 is defined a-priori through a noise schedule. We train a denoiser network, to reverse this process given the timestep 𝑡 , the instruction 𝐿 , the noised target motion 𝑀 𝑡 𝑇 and the source motion 𝑀 𝑆 . As supervision, the output of the denoiser network ˜ 𝑀 𝑡 𝑇 is compared against the ground-truth denoised target motion 𝑀 𝑇 . Our model is therefore trained to minimize:

$$\mathbb { E } _ { \epsilon \sim \mathcal { N } ( 0, 1 ), t, L, M _ { S } } \left \| D \left ( M _ { T } ^ { t } ; t, L, M _ { S } \right ) - M _ { T } \right \| \quad ( 1 )$$

We use standard mean-squared-error as the loss function to compare the diffusion output with the ground-truth target motion. We choose to predict the denoised target motion, as we found this to produce better results visually than predicting the noise itself.

The architecture overview is illustrated in Figure 3 (left). Our model consists of multiple encoders for each input modality ( 𝐸 𝑇 for timestep, 𝐸 𝐿 for text, and 𝐸 𝑀 for motion) and a transformer encoder 𝐷 that operates on all inputs. The timestep 𝑡 is encoded via 𝐸 𝑇 similar to MDM [Tevet et al. 2023], by first converting into a sinusoidal positional embedding, and then projecting through a feed-forward network (consisting of two linear layers with a SiLU activation [Elfwing et al. 2018] in between). As in [Tevet et al. 2023], we use the CLIP [Radford et al. 2021] text encoder for 𝐸 𝐿 . We pass the source and noised target motions through a linear layer ( 𝐸 𝑀 ), shared across frames, and obtain 𝑀 𝑒𝑛𝑐 𝑆 = 𝐸 𝑀 ( 𝑀 𝑆 ) and 𝑀 𝑡 𝑒𝑛𝑐 𝑇 =

𝐸 𝑀 ( 𝑀 𝑡 𝑇 ) . Given the variable duration of source and target motions, we add a learnable separation token SEP in between [Devlin et al. 2019] when appending them (so that the information on when the target motion ends and the source motion starts is communicated to the transformer). Once all encoded inputs have the same feature dimensionality 𝑑 , they are combined into a single sequence to be fed to the transformer, as shown in Figure 3, and sinusoidal positional embeddings are subsequently added. During training, to enable classifier-free guidance, the source motion condition is randomly dropped 5% of the time, the text condition 5%, both conditions together 5%, and all the inputs are used 85% of the time. For sampling from a diffusion model with two conditions, we apply classifier-free guidance with respect to two conditions: the input motion 𝑀 𝑆 and the text instruction 𝐿 . We introduce separate guidance scales 𝑠 𝑀 𝑆 and 𝑠 𝐿 that allow adjusting the influence of each conditioning.

For simplicity, we now abuse the notation by dropping the timestep subscripts when deriving the sampling process. Our generative model, TMED, learns the probability distribution over the target motions, 𝑀 𝑇 , conditioned on the source motions and text condition, 𝑃 ( 𝑀 𝑇 | 𝑀 ,𝐿 𝑆 ) . Expanding this conditional probability gives:

$$\Lambda \cdots \alpha, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \LambDA, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lamb Da, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lamboda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambma, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambd, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \LambDa, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lamb D, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \Lambda, \LambD, \Lambda, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambdD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LembD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LAMD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LabD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LamD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LABD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LIBD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LMBD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LmbD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LimbD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \LambD, \L$$

As in the original diffusion, we formulate this as a score function optimization problem by first taking the logarithm of Eq.(2):

$$\int \liminf _ { P \text{VOC} } \int \liminf _ { \max } \longrightarrow \int \liminf _ { \max } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longleftarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int\liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _{ \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \longrightarrow \int \liminf _ { \min } \\ \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min} \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min. } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \max } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \sum } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \ min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \ max } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \infty } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \min } \int \liminf _ { \begin{array} { c } } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \\ \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \int \liminf _ { c } \end{array}$$

Then, the derivative with respect to the input of Eq.(3) gives the score estimate ˜ 𝑒 𝜃 ( 𝑀 ,𝑠 𝑇 𝑀 𝑆 , 𝑠 𝐿 ) , learned under classifier-free guidance:

$$\nabla _ { M _ { T } } ^ { \vee \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \colon } \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \eta \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \zi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \Xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xrightarrow \, \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \ \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \, \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cd. \end{cases} \frac { \mu \nu } \lambda \lambda } \lambda } \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda } } \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda
 } } \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \xi \Xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \xi \tag \tag \, \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cd. \quad \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \dots } \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda. } } \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \lambda \tag \, \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot
 \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot \cdot$$

Hence, from Eq.(4), we sample from TMED using the modified score estimate of two-way conditioning a diffusion model as:

$$\tilde { e } _ { \theta } ( M _ { T }, s _ { M _ { S } }, s _ { L } ) & = e _ { \theta } ( M _ { T }, \emptyset, \overset { \searrow } { \emptyset } ) & \quad \text{indic} \\ & \quad + s _ { M _ { S } } \cdot ( e _ { \theta } ( M _ { T }, M _ { S }, \emptyset ) - e _ { \theta } ( M _ { T }, \emptyset, \emptyset ) ) & \quad ( 5 ) & \quad \text{in} \, \text{Se p} \\ & \quad + s _ { L } \cdot ( e _ { \theta } ( M _ { T }, M _ { S }, L ) - e _ { \theta } ( M _ { T }, M _ { S }, \emptyset ) ). & \quad \text{conf}$$

We further ablate the guidance scales which control the generation at test time in Section 5.

Implementation details. All models are trained for 1000 epochs using cosine noise schedule with DDPM scheduler [Ho et al. 2020]. We use 𝑁 = 300 diffusion timesteps, as we find this is a good compromise between speed and quality. The guidance scales are chosen for each model based on their best performance in the validation set of MotionFix ( 𝑆 𝐿 = 2 , 𝑠 𝑀 𝑆 = 2). We follow the same process for training MDM [Tevet et al. 2023] baselines described in the next section. In terms of architectural details, the dimensionality of the embeddings before inputting to the transformer is 𝑑 = 512. We use a pre-trained and frozen CLIP [Radford et al. 2021] with all 77 token outputs of the ViT-B/32 backbone [Dosovitskiy et al. 2021] as our text encoder 𝐸 𝐿 . We use the text masks from 𝐸 𝐿 to mask the padded area of the text inputs. The motion encoder 𝐸 𝑀 that precedes the transformer is a simple linear projection with dimensionality 𝑑 𝑝 × 𝑑 , where the feature dimension of each motion frame is 𝑑 𝑝 = 207 (as described in Section 4.1).

## 5 EXPERIMENTS

Westart by describing our evaluation metrics for the new MotionFix benchmark (Section 5.1). We then present the main results on this task, comparing our proposed model to our baseline designs (Section 5.2). Next, we provide ablations on the training data size and guidance hyperparameters (Section 5.3). Finally, we demonstrate qualitative results, comparisons with the baselines and samples from our dataset (Section 5.4).

## 5.1 Evaluation Metrics

Similar to text-to-motion synthesis, distance-based metrics to evaluate motion generation quality is problematic due to multiple plausible ground-truth motions for a given text. Prior work has extensively used text-to-motion retrieval metrics for evaluating text-to-motion synthesis [Guo et al. 2022b; Tevet et al. 2023], by training a textmotion contrastive model and using its features. To evaluate motion editing, we introduce motion-to-motion retrieval metrics. Given a generated motion, we measure how well the source ( generatedto-source retrieval ) or the target motion ( generated-to-target retrieval ) can be retrieved. We use TMR [Petrovich et al. 2023] as the feature extractor, but train it ourselves to support our feature representation, using the same regime as in the original paper with HumanML3D data [Guo et al. 2022a]. We report standard metrics, R@1, R@2, R@3 and AvgR using a gallery size of 32 randomly sampled batches for retrieval from the test set. Recall at rank 𝑘 (R@k) computes the percentage of times the correct motion is among the top 𝑘 results. Note that we fix the batches so there is no randomness across evaluations. The performance is averaged across batches. We report results on the full test set as gallery in Tables A.2 and A.3 of the appendix, where the same conclusions hold. While the main performance measure is according to generated-to-target retrieval, we also monitor how close our generations remain to the source. As indicative values, we provide ground-truth (GT) values for the latter. We provide additional measures (FID, L2) and perceptual studies in Sections B and C of the appendix, respectively, which further confirm the results of our proposed retrieval-based benchmarking.

## 5.2 Comparison to Baselines

We report our main results in Table 2. We compare the performance of TMED trained on our MotionFix triplets against several baselines trained on the larger, HumanML3D text-motion pairs [Guo et al. 2022a]. We build our baselines by training MDM [Tevet et al. 2023] with our human motion representation and by repurposing this model for motion editing, described next.

Wefirst introduce two simple baselines: (a) MDM that purely uses the edit text as input to the text-to-motion generation (i.e., without a source motion), and (b) MDM 𝑆 that additionally uses the source

•

7

| Methods   | Data      | Source input   | generated-to-target retrieval   | generated-to-target retrieval   | generated-to-target retrieval   | generated-to-target retrieval   | generated-to-source retrieval   | generated-to-source retrieval   | generated-to-source retrieval   | generated-to-source retrieval   |
|-----------|-----------|----------------|---------------------------------|---------------------------------|---------------------------------|---------------------------------|---------------------------------|---------------------------------|---------------------------------|---------------------------------|
| Methods   | Data      | Source input   | R@1                             | R@2                             | R@3                             | AvgR                            | R@1                             | R@2                             | R@3                             | AvgR                            |
| GT        | n/a       | n/a            | 100.0                           | 100.0                           | 100.0                           | 1.00                            | 74.01                           | 84.52                           | 89.91                           | 2.03                            |
| MDM       | HumanML3D | ✗              | 4.03                            | 7.56                            | 10.48                           | 15.55                           | 2.62                            | 6.15                            | 9.38                            | 15.88                           |
| MDM 𝑆     | HumanML3D | ✓ , init       | 3.63                            | 7.06                            | 10.08                           | 15.64                           | 2.62                            | 6.25                            | 9.78                            | 15.84                           |
| MDM-BP 𝑆  | HumanML3D | ✓ , init&BP    | 38.10                           | 48.99                           | 54.84                           | 6.47                            | 60.28                           | 69.46                           | 73.89                           | 4.23                            |
| MDM-BP    | HumanML3D | ✓ , BP         | 39.10                           | 50.09                           | 54.84                           | 6.46                            | 61.28                           | 69.55                           | 73.99                           | 4.21                            |
| TMED      | MotionFix | ✓ , condition  | 62.90                           | 76.51                           | 83.06                           | 2.71                            | 71.77                           | 84.07                           | 89.52                           | 1.96                            |

Table 2. Results on the MotionFix benchmark (test set): We first evaluate several variants of our text-to-motion synthesis baseline (MDM) on the motion editing task. Subscript 𝑆 denotes models that denoise the source motion initialization (init) instead of starting the diffusion from noise. BP indicates GPT-based body part labeling described in Section 4 to mask the source body parts which are kept unchanged during diffusion. Our model TMED effectively learns how to utilize the source motion conditioning, thanks to the MotionFix training data. See text for detailed comments.

| Methods   | generated-to-target retrieval   | generated-to-target retrieval   | generated-to-target retrieval   | generated-to-target retrieval   | generated-to-source retrieval   | generated-to-source retrieval   | generated-to-source retrieval   | generated-to-source retrieval   |
|-----------|---------------------------------|---------------------------------|---------------------------------|---------------------------------|---------------------------------|---------------------------------|---------------------------------|---------------------------------|
| Methods   | R@1                             | R@2                             | R@3                             | AvgR                            | R@1                             | R@2                             | R@3                             | AvgR                            |
| GT        | 100.0                           | 100.0                           | 100.0                           | 1.00                            | 74.01                           | 84.52                           | 89.91                           | 2.03                            |
| 10%       | 19.25                           | 30.65                           | 38.71                           | 8.92                            | 22.98                           | 37.50                           | 45.97                           | 7.50                            |
| 50%       | 47.08                           | 61.49                           | 69.66                           | 4.23                            | 54.44                           | 70.06                           | 78.12                           | 3.33                            |
| 100%      | 62.90                           | 76.51                           | 83.06                           | 2.71                            | 71.77                           | 84.07                           | 89.52                           | 1.96                            |

Table 3. Effect of training data size in MotionFix: Weobserve significant performance improvement as we increase the amount of training data.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-afb3bc3c-4542-4333-8511-05e06a22339e.jpg)

motion as input instead of noise during inference. For the latter, we also investigated reducing the number of diffusion steps when initializing from source; however, we observed performance drops and therefore kept the full 300 diffusion steps. Inspired by [Athanasiou et al. 2023], we design two additional strong baselines ( MDM-BP 𝑆 and MDM-BP ), that are based on body part labels extracted by querying GPT with the edit texts. We automatically detect body parts which are irrelevant to the text and keep them constant via masking. We again initialize the diffusion process either from the source motion (MDM-BP 𝑆 ) or from noise (MDM-BP) for the body parts that need to change according to the GPT response. For more details on the query and example GPT outputs, we refer to Section D of the appendix.

Wefirst observe from Table 2 that, for all the baselines, initializing from noise performs better than initializing from source motion. Our strong baselines based on body-part detection (MDM-BP, MDMBP ) clearly outperform the naive baselines. However, all baselines 𝑆 fall behind our TMED that successfully leverages the access to training triplets, and significantly outperforms alternatives.

Moreover, MDM-BP, MDM-BP 𝑆 are both strong baselines, but relying on GPT body part labels might not capture all edit types, such as the ones that require modifying the overall body. We demonstrate this further in our supplementary video from the project webpage and our qualitative comparisons (Section 5.4).

## 5.3 Ablations

In the following, we investigate the effect of training data size and the guidance scales on the TMED model performance.

Fig. 4. Guidances of conditions: We illustrate the R@1 performance of TMEDforgenerated-to-target (left) and generated-to-source (right) retrieval benchmarks for 𝑠 𝐿 , 𝑠 𝑀𝑆 ∈ [ 1 5 , ] .

Training data size. In Table 3, we present the performance of TMED for different data sizes from MotionFix. We clearly observe, that increasing the data size has a large impact on the performance, justifying our data collection. The non-saturated trend is encouraging to scale up the training further.

Guidance hyperparameters. In Figure 4, we present how TMED performs across different guidance values for both conditions. 𝑥 -axis controls the text guidance 𝑠 𝐿 , and 𝑦 -axis controls the source motion guidance 𝑠 𝑀 𝑆 at test time. We report both generated-totarget (left) and source-to-target (right) R@1 retrieval results. We observe that there needs to be a balance between the two guidance values, and that performances decrease towards the extremes (e.g., top left and bottom right corners of the plots, where only one of the two conditions have higher guidance). This highlights the need to rely on both conditions to perform the task.

## 5.4 Qualitative Results

We display several generations from TMED in Figure 5 to enable qualitative assessment. We observe that our model can perform different types of edits such as the addition of actions ('rotate wrists instead of stretching like yawn'), temporal edits in a motion ('get up a bit earlier'), speed edits ('slow down') and combinations of these. We refer to our supplementary video for dynamic visualizations, which may be easier to interpret.

Fig. 5. TMED generations: We illustrate several generations from our model with overlaid source (red) and generated (blue) motions. We showcase a variety of test cases ranging from elaborate edits (first example in top left) to short commands (e.g., 'mirror'). TMED is able to perform both edits that describe temporal (e.g., 'slow down') or spatial (e.g., 'raise your arms higher so it is overhead') modifications.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-90c75ea8-cbe5-4c3c-971e-ad4702a3a811.jpg)

In Figure 6, we further provide examples of failures cases from TMED. In the top row, we analyze cases with long edit texts. The model struggles with complex details and does not 'keep the body straight' in the left example, nor follows 'bend arms in the elbows' instructions on the right side, while wider legs are correctly edited. In the bottom row, we illustrate examples where the model faithfully follows the edit text, but does not resemble the source motion. In the left generation, the steps are correctly wider, but the movement does not continue to the similar position as the source motion. Finally, on the right, the body is kneeling down faster as instructed, but towards the opposite direction.

In the next two rows, we visualize MDM-BP 𝑆 results. Given the text 'rotate wrists instead of stretching like yawn', GPT correctly suggests editing both hands; however, the generated motion no longer resembles the source motion as the wide-open hands are not preserved. For the example edit 'turn in the opposite direction', all body parts are involved, but MDM-BP 𝑆 does not deviate too much from the source, perhaps because traditional text-to-motion generation models rarely see relative words such as 'opposite direction'.

We additionally provide a qualitative comparison in Figure 7, between TMED and various baselines. We provide two comparisons for each baseline (top block for MDM 𝑆 , middle block for MDM-BP 𝑆 , and bottom block for MDM-B).

We observe that MDM 𝑆 picks up the action from the prompt, but fails to faithfully follow the source motion. In the first row, the generation by MDM 𝑆 raises both hands, instead of adjusting only the height of the hand raised in the source motion. Similarly, in the second row, MDM 𝑆 generation raises the arm but in front of the body and not higher as prompted by the edit text.

Finally, we illustrate generations from our strongest baseline MDM-BP in the last two rows. Both generations involve all parts of the body (e.g., 'slow down') making it hard to follow the source motion. In comparison, our model faithfully performs most of the edits. The disadvantage of TMED, on the other hand, might be the generalization to motion pairs where the TMR similarity is low as such edits were unseen during training. We briefly discuss more limitations in the following.

## 6 CONCLUSIONS

In this work, we studied the task of motion editing from language instructions. Given the scarcity of training data, we introduced a new dataset MotionFix, collected in a semi-automatic manner. We exploit motion retrieval models to obtain 'edit-ready' motion pairs which

Fig. 6. Failure cases: We show four failure examples from our model. For each sample, we provide the source motion (red) overlaid both with the generation (blue, left) or the ground-truth target motion (green, right). In the top row, we observe that the model may fail to generate the edited motions when the edit text is detailed and the motions differences are subtle. In the bottom row, although the generated motions follow the edit text, they diverge from the source motions.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-41ebb7a4-a1b6-4d5f-b18f-12873ebd03a8.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-d59cf3a5-1ed4-4855-ba4f-6ba40b959bf3.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-73d72d32-cccb-438f-94cf-cbf16e5e1937.jpg)

we annotate with language labels. We design a conditional diffusion model TMED that is trained on MotionFix, and generates edited motions that follow the source motion and the edit text. We show both quantitatively and qualitatively that our model outperforms all baselines. We hope that our dataset and findings will assist the research community and pave the way for exploring this new task. Limitations. Our approach comes with limitations. Assuming two TMR-similar motions being an editing distance apart is not always accurate but serves as a good starting point. Furthermore, in our data collection, we constrain the motions to be up to 5 seconds since longer motions to produce many dissimilar pairs. Regarding model performance, TMED exhibits difficulty generalizing to unseen or complex edit texts and maintaining faithfulness to the source motion. Moreover, while our model can be used iteratively, we do not explore this capability in this paper and leave for future work.

## ACKNOWLEDGMENTS

The authors would like to thank Benjamin Pellkofer for building the data exploration website, Tsvetelina Alexiadis for guidance in data collection and perceptual studies, Arina Kuznetcova, Asuka Bertler, Claudia Gallatz, Suraj Bhor, Tithi Rakshit, Taylor McConnell, Tomasz Niewiadomski for data annotation, Lea Müller and Mathis Petrovich for helpful discussions, Yuliang Xiu for proofreading and Peter Kulits for the support and seatmating. GV acknowledges the ANR project CorVis ANR-21-CE23-0003-01. Disclosure: https:// files.is.tue.mpg.de/black/CoI\_CVPR\_2024.txt

## REFERENCES

Kfir Aberman, Yijia Weng, Dani Lischinski, Daniel Cohen-Or, and Baoquan Chen. 2020. Unpaired motion style transfer from video to animation. Transactions on Graphics (TOG) (2020).

- Nikos Athanasiou, Mathis Petrovich, Michael J Black, and Gül Varol. 2022. TEACH: Temporal Action Composition for 3D Humans. In International Conference on 3D Vision (3DV) .
- Nikos Athanasiou, Mathis Petrovich, Michael J. Black, and Gül Varol. 2023. SINC: Spatial Composition of 3D Human Motions for Simultaneous Action Generation. International Conference on Computer Vision (ICCV) (2023).
- Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J. Black. 2016. Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image. In European Conference on Computer Vision (ECCV) .
- Tim Brooks, Aleksander Holynski, and Alexei A Efros. 2023. InstructPix2Pix: Learning to follow image editing instructions. In Computer Vision and Pattern Recognition (CVPR) .
- Edwin Catmull. 1972. A system for computer generated movies. In Proceedings of the ACM Annual Conference .
- Setareh Cohan, Guy Tevet, Daniele Reda, Xue Bin Peng, and Michiel van de Panne. 2024. Flexible Motion In-betweening with Diffusion Models. In International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH) .
- Michael F Cohen. 1992. Interactive spacetime control for animation. In International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH) .
- Ginger Delmas, Philippe Weinzaepfel, Thomas Lucas, Francesc Moreno-Noguer, and Grégory Rogez. 2022. PoseScript: 3D Human Poses from Natural Language. In European Conference on Computer Vision (ECCV) .
- Ginger Delmas, Philippe Weinzaepfel, Francesc Moreno-Noguer, and Grégory Rogez. 2023. PoseFix: Correcting 3D Human Poses with Natural Language. In International Conference on Computer Vision (ICCV) .
- J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pretraining of Deep Bidirectional Transformers for Language Understanding. In Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) .
- Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations (ICLR) .
- Stefan Elfwing, Eiji Uchibe, and Kenji Doya. 2018. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks (2018).
- Mihai Fieraru, Mihai Zanfir, Silviu Cristian Pirlea, Vlad Olaru, and Cristian Sminchisescu. 2021. AIFit: Automatic 3D human-interpretable feedback models for fitness training. In Computer Vision and Pattern Recognition (CVPR) .

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-611f1cd2-b4a8-477d-b3ae-91ec222cc815.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-5aaf1929-e6f1-43c0-b3df-5a748cfa0f71.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-6511fd80-f68d-4b78-ae28-2dd54309e084.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-08ccc77b-1d70-4e1a-a889-028f099c5bf5.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-c9960235-f070-44c2-bb2b-d566f721458e.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-e9bb5bcc-84bf-4dc6-a4d3-4afe115f2634.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-c0928f1d-ad62-4cae-b960-c5851d93e3d3.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-09aa95f5-a2a6-4a63-a46b-6dfb1e4dedc9.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-1b60e816-239e-4ee4-aba2-3e61c8a09906.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-702e5ddf-0b1f-48ab-9dee-23cf4123d94e.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-42650d42-8ba2-4796-96a8-3f1a4408a652.jpg)

TMED vs MDM

S

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-d0128f9f-bdd5-49b1-9c49-6614ea59a48d.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-a43bceef-a728-4947-a459-e889f061643b.jpg)

do it with the hand raised lower

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-20a2ea94-f32b-4e20-b8c4-dbd507f51f7a.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-0f2869d1-943a-4e08-b1a2-db0a18d66ec5.jpg)

raise arm up a little more and give a light shake

TMED vs MDM-BP S

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-eb2a1e77-a759-41e5-9dc9-e79c0d7e1a78.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-a912f4a7-87e7-47b8-b155-132cbc908422.jpg)

rotate wrists instead of stretching like yawn

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-47ec27d8-b555-4ddf-ab97-9cfea3ffeb4f.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-4c455e95-6aff-4bf7-b2b5-b8467887c03a.jpg)

turn in opposite direction

TMED vs MDM-BP

slow down

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-da27d036-bae1-4a35-ab6e-22c60919aef3.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-8981256e-4c53-4fcb-9b51-fa1823dc3554.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-53d7efc1-05c6-4c6d-9476-3f0300c61cf6.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-12faea44-2359-46aa-8e4a-4f738cc1ae7b.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-011cb7cb-3921-4805-bcc4-5ede83f5b687.jpg)

donGLYPH&lt;213&gt;t get up

Fig. 7. Qualitative comparisons with baselines: We provide example results by comparing TMED against the baselines on the MF test set: MDM 𝑆 (top), MDM-BP 𝑆 (middle) and MDM-BP (bottom). First two columns show the source (red) and ground-truth target (green) motions. Third column is reserved for baselines, the last column for TMED. Generations are denoted in blue.

Purvi Goel, Kuan-Chieh Wang, C Karen Liu, and Kayvon Fatahalian. 2024. Iterative Motion Editing with Natural Language. In International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH) .

- Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. 2024. MoMask: Generative Masked Modeling of 3D Human Motions. In Computer Vision and Pattern Recognition (CVPR) .
- Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, and Ji. 2022a. Generating diverse and natural 3D human motions from text. In Computer Vision and Pattern Recognition (CVPR) .
- Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. 2022b. TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts. In European Conference on Computer Vision (ECCV) .
- Chuan Guo, Xinxin Zuo, Sen Wang, Shihao Zou, and Sun. 2020. Action2Motion: Conditioned Generation of 3D Human Motions. In ACM International Conference on Multimedia (MM) .

Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020. Denoising Diffusion Probabilistic Models. In Conference on Neural Information Processing Systems (NeurIPS) .

Daniel Holden, Jun Saito, and Taku Komura. 2016. A Deep Learning Framework for Character Motion Synthesis and Editing. Transactions on Graphics (TOG) (2016).

Yiming Huang, Weilin Wan, Yue Yang, Chris Callison-Burch, Mark Yatskar, and Lingjie Liu. 2024. CoMo: Controllable Motion Generation through Language Guided Pose Code Editing. arXiv:2403.13900 (2024).

Korrawe Karunratanakul, Konpat Preechakul, Supasorn Suwajanakorn, and Siyu Tang. 2023. Guided Motion Diffusion for Controllable Human Motion Synthesis. In Computer Vision and Pattern Recognition (CVPR) .

Hyounghun Kim, Abhay Zala, Graham Burri, and Mohit Bansal. 2021. FixMyPose: Pose Correctional Captioning and Retrieval. AAAI Conference on Artificial Intelligence (2021).

- Jihoon Kim, Jiseob Kim, and Sungjoon Choi. 2023. FLAME: Free-form language-based motion synthesis &amp; editing. In AAAI Conference on Artificial Intelligence .
- Diederik P Kingma and Max Welling. 2014. Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR) .
- Makito Kobayashi, Chen-Chieh Liao, Keito Inoue, Sentaro Yojima, and Masafumi Takahashi. 2023. Motion capture dataset for practical use of AI-based motion editing and stylization. arXiv:2306.08861 (2023).

Jehee Lee and Sung Yong Shin. 1999. A hierarchical approach to interactive motion editing for human-like figures. In International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH) .

- Taeryung Lee, Gyeongsik Moon, and Kyoung Mu Lee. 2022. MultiAct: Long-Term 3D Human Motion Generation from Multiple Action Labels. In AAAI Conference on Artificial Intelligence .

Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. 2023. Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset. In Conference on Neural Information Processing Systems (NeurIPS) .

Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. 2015. SMPL: A Skinned Multi-Person Linear Model. Transactions on Graphics (TOG) (2015).

Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. 2019. AMASS: Archive of Motion Capture As Surface Shapes. In International Conference on Computer Vision (ICCV) .

Ian Mason, Sebastian Starke, and Taku Komura. 2022. Real-time style modelling of human locomotion via feature-wise transformations and local motion phases. Proceedings of the ACM on Computer Graphics and Interactive Techniques (i3D) (2022).

Ken Perlin. 1995. Real time responsive animation with personality. IEEE Transactions on Visualization and Computer Graphics (1995).

Mathis Petrovich, Michael J Black, and Gül Varol. 2021. Action-conditioned 3D human motion synthesis with Transformer VAE. In International Conference on Computer Vision (ICCV) .

Mathis Petrovich, Michael J Black, and Gül Varol. 2022. TEMOS: Generating diverse human motions from textual descriptions. In European Conference on Computer Vision (ECCV) .

Mathis Petrovich, Michael J. Black, and Gül Varol. 2023. TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesis. In International Conference on Computer Vision (ICCV) .

Mathis Petrovich, Or Litany, Umar Iqbal, Michael J. Black, Gül Varol, Xue Bin Peng, and Davis Rempe. 2024. Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation. In Computer Vision and Pattern Recognition Workshops (CVPRW) . Matthias Plappert, Christian Mandery, and Tamim Asfour. 2016. The KIT Motion-

Language Dataset. Big Data (2016).

Zoran Popović and Andrew Witkin. 1999. Physically based motion transformation. In International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH) .

Abhinanda R. Punnakkal, Arjun Chandrasekaran, Nikos Athanasiou, Alejandra QuirosRamirez, and Michael J. Black. 2021. BABEL: Bodies, Action and Behavior with English Labels. In Computer Vision and Pattern Recognition (CVPR) .

Alec Radford, Jong Wook Kim, Chris Hallacy, and Ramesh. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML) .

- Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Computer Vision and Pattern Recognition (CVPR) .

Yonatan Shafir, Guy Tevet, Roy Kapon, and Bermano. 2024. Human Motion Diffusion as a Generative Prior. In International Conference on Learning Representations (ICLR) .

- Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. 2015. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning (ICML) .
- Sebastian Starke, He Zhang, Taku Komura, and Jun Saito. 2019. Neural state machine for character-scene interactions. Transactions on Graphics (TOG) (2019).
- Guy Tevet, Brian Gordon, Amir Hertz, Amit H Bermano, and Daniel Cohen-Or. 2022. MotionCLIP: Exposing human motion generation to clip space. In European Conference on Computer Vision (ECCV) .
- Guy Tevet, Sigal Raab, Brian Gordon, and Shafir. 2023. Human Motion Diffusion Model. In International Conference on Learning Representations (ICLR) .
- Kengo Uchida, Takashi Shibuya, Yuhta Takida, Naoki Murata, Shusuke Takahashi, and Yuki Mitsufuji. 2024. MoLA: Motion Generation and Editing with Latent Diffusion Enhanced by Adversarial Training. arXiv:2103.15691 (2024).
- Weilin Wan, Yiming Huang, Shutong Wu, Taku Komura, Wenping Wang, Dinesh Jayaraman, and Lingjie Liu. 2023. DiffusionPhase: Motion Diffusion in Frequency Domain. arXiv:2312.04036 (2023).
- Yiming Xie, Varun Jampani, Lei Zhong, Deqing Sun, and Huaizu Jiang. 2024. OmniControl: Control Any Joint at Any Time for Human Motion Generation. In International Conference on Learning Representations (ICLR) .
- Mingyuan Zhang, Zhongang Cai, Liang Pan, and Hong. 2023a. MotionDiffuse: Textdriven human motion generation with diffusion model. Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (2023).
- Mingyuan Zhang, Huirong Li, Zhongang Cai, Jiawei Ren, Lei Yang, and Ziwei Liu. 2023b. FineMoGen: Fine-Grained Spatio-Temporal Motion Generation and Editing. In Conference on Neural Information Processing Systems (NeurIPS) .
- Qinsheng Zhang, Jiaming Song, Xun Huang, and Chen. 2023c. DiffCollage: Parallel generation of large content with diffusion models. In Computer Vision and Pattern Recognition (CVPR) .
- Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. 2019. On the Continuity of Rotation Representations in Neural Networks. In Computer Vision and Pattern Recognition (CVPR) .
- Qiran Zou, Shangyuan Yuan, Shian Du, Yu Wang, Chang Liu, Yi Xu, Jie Chen, and Xiangyang Ji. 2024. ParCo: Part-Coordinating Text-to-Motion Synthesis. ECCV (2024).

## APPENDIX

This document provides additional details about our dataset statistics (Section A), additional quantitative evaluations (Section B), results of our perceptual studies (Section C), and our GPT-based body-part annotation for the baseline (Section D).

Supplementary video. Along with this document, we provide a video on our webpage, which includes visualizations of motions from the MotionFix dataset and comparisons with baselines. In specific: (i) We first briefly describe our goal. (ii) We then show examples from the dataset we collected for different types of edits. (iii) We provide additional qualitative results showing the capability of our method to generate different types of edits. (iv) We provide qualitative comparisons against baselines, highlighting their main problems and how our method overcomes them.

## A MOTIONFIX STATISTICS

We provide additional information about our MotionFix dataset described in Section 3 of the main paper. Since our task is about editing rather than describing motion, it is important that the edits in our dataset contain phraseology of that kind. To confirm this, in Figure A.1 we present qualitative and quantitative visualizations of the word statistics or our dataset's edit texts. Aside from words that describe body parts, we observe that many of the popular words have an inherently comparative nature. For example, words such as 'keep', 'instead', 'slower', 'faster', 'closer', 'earlier', 'right', 'left', are frequently used as shown in the word cloud visualization. Additionally, in Table A.1 we provide statistics related to the annotated text describing a motion edit. Notice that MotionFix contains a wide spectrum of text descriptions of motion edits ranging from single word commands (e.g., 'slower') to larger paragraphs that describe edits of multiple body parts at different time segments of the motion. Moreover, MotionFix contains 4649, 4165 unique source and target motions, respectively.

## B ADDITIONAL QUANTITATIVE EVALUATIONS

Using the entire test set as gallery. In addition to Tables 2 and 3 of the main paper, for completeness, we provide evaluations that use the entire test set as a gallery in Tables A.2 and A.3. Since in this setup, models need to retrieve the target or source motion from a larger pool of motions, it is expected to score lower recall. In terms of relative improvements and conclusions, we still observe the same trends between Table A.2 and Table 2 (main paper), as well as between Table A.3 and Table 3 (main paper).

Additional metrics. To provide a more comprehensive evaluation, besides our retrieval-based metrics, we compute additional measures: FID of motion features and L2 distance between joint positions. We compute both between the test set of MotionFix and the generated motions. For FID, we use the motion branch from the TMR model for the computation of the retrieval metrics. TMED has an FID of 0 129 outperforming the two best baselines 0 152 for . . MDM-BP 𝑆 and 0 145 for MDM-BP. While TMED matches best the . distribution of target motions, the baselines perform closely indicating that all models maintain certain degree of realism. Our model also has a lower L2 distance (TMED: 1.10cm, MDM-BP 𝑆 : 1.26cm,

## Text in MotionFix triplets

Table A.1. Statistics of the MotionFix textual data: There are relatively low number of duplicate texts (given 6730 triplets and 5992 unique texts). The vocabulary is diverse ( 1479 unique words) and the average number of words per text ( 8 46 . ) has a good trade-off between conciseness and expressiveness.

| Total #triplets        |   6730 |
|------------------------|--------|
| #Unique texts          | 5992   |
| #Unique words          | 1479   |
| Avg #words per text    |    8.5 |
| Median #words per text |    8   |
| Std of #words per text |    4.9 |
| Min #words per text    |    1   |
| Max #words per text    |   43   |

MDM-BP: 1.22cm). Note that coordinate-based metrics are not always reliable given multiple plausible generations due to language ambiguity.

## C PERCEPTUAL STUDIES

To further evaluate TMED, we conducted two perceptual studies: the first one evaluating the absolute quality using Likert scale, the second one performing a comparative evaluation. In both studies, we used MDM-BP as the strongest baseline to compare with. We conducted the studies across 25 workers and averaged the results. We used 150 randomly chosen videos from the test set, with each study using a random subset of 50 videos for each method, from those 150 videos. For both studies, the source motion and generation (or ground-truth) were rendered overlaid.

Likert scale: absolute quality. The objective of this study is to rate how well the instructions are reflected in the videos using a 5-point Likert scale, by asking users to rate how much they agree with the statement, 'the green (generated or ground-truth) motion follows the instruction given the red (source) motion.' Choices range from '5: completely agree' to '1: completely disagree.' The workers were presented with videos from 3 sets of motions: ground-truth, TMED, and MDM-BP. Each subset is represented by 50 videos, making a total of 150 videos. The videos are presented in 3 batches, each containing a randomized selection from the three sets. Each batch has 17 videos from each set, totaling 51 videos per batch. This ensures that workers evaluate a balanced mix of videos from all three in each batch. The ground-truth (GT) motions are ranked first, TMED second, and MDM-BP third, with means and standard deviations of 𝜇 ± 𝜎 : 3.93 ± 0.95 (GT), 3.59 ± 1.15 (TMED), and 3.52 ± 1.24 (MDM-BP). This verifies our conclusions in Table 2 (main paper) and Table A.2.

Pairwise comparison: relative quality. The objective of the second study is to compare our model (TMED) against the best baseline (MDM-BP) by displaying them side-by-side and asking which motion generation better follows the instructions. This study involves a total of 50 videos, divided into two batches of 25 videos each. Additionally, each batch includes 5 initial comparisons to familiarize participants with the task, which are randomly repeated later and discarded from the computations. Moreover, we include 3 catch

count

1750

1500

1250

1000

750

500

250

0

•

30 most common words

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-33f2ec86-4d8e-471d-b053-971e2283e791.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00712v3-figure-31373d81-bfdf-476f-8c17-68d47f50deba.jpg)

Fig. A.1. Word frequencies in the MotionFix dataset: On the left, we display a word cloud for the text annotations in in the dataset. Most frequent words appear in larger fonts. Examples of such words are 'hand', 'arm' referring to body parts, 'instead' referring to the source motion, 'higher', 'lower', 'opposite', 'slower' referring to spatial, directional or speed edits. On the right, we show the histogram of the 30 most frequent words in the data.

| Methods   | Data      | Source input   | generated-to-target retrieval   | generated-to-target retrieval   | generated-to-target retrieval   | generated-to-target retrieval   | generated-to-source retrieval   | generated-to-source retrieval   | generated-to-source retrieval   | generated-to-source retrieval   |
|-----------|-----------|----------------|---------------------------------|---------------------------------|---------------------------------|---------------------------------|---------------------------------|---------------------------------|---------------------------------|---------------------------------|
| Methods   | Data      | Source input   | R@1                             | R@2                             | R@3                             | AvgR                            | R@1                             | R@2                             | R@3                             | AvgR                            |
| GT        | n/a       | n/a            | 64.36                           | 88.75                           | 95.56                           | 1.74                            | 20.83                           | 33.66                           | 40.47                           | 33.13                           |
| MDM       | HumanML3D | ✗              | 0.00                            | 0.30                            | 0.49                            | 476.65                          | 0.10                            | 0.10                            | 0.10                            | 486.26                          |
| MDM 𝑆     | HumanML3D | ✓ , init       | 0.00                            | 0.10                            | 0.10                            | 477.68                          | 0.00                            | 0.10                            | 0.30                            | 485.02                          |
| MDM-BP 𝑆  | HumanML3D | ✓ , init&BP    | 8.39                            | 14.81                           | 18.36                           | 181.38                          | 30.11                           | 36.72                           | 40.77                           | 107.11                          |
| MDM-BP    | HumanML3D | ✓ , BP         | 8.69                            | 14.71                           | 18.36                           | 180.99                          | 30.21                           | 36.82                           | 40.47                           | 106.05                          |
| TMED      | MotionFix | ✓ , condition  | 14.51                           | 21.72                           | 28.73                           | 56.63                           | 22.41                           | 34.45                           | 40.57                           | 31.42                           |

Table A.2. Results on the MotionFix benchmark using the whole test set as a gallery: We evaluate the models in Table 2 of the main paper on the full test set of 1013 samples (as opposed to a random subset of 32). While the retrieval metrics are unsurprisingly lower due to a larger gallery size, the conclusions hold.

Table A.3. Text-based motion editing benchmark on MotionFix test set with different training data sizes. We observe that the performance increases significantly when more data are used during training.

| Methods   | generated-to-target retrieval   | generated-to-target retrieval   | generated-to-target retrieval   | generated-to-target retrieval   | generated-to-source retrieval   | generated-to-source retrieval   | generated-to-source retrieval   | generated-to-source retrieval   |
|-----------|---------------------------------|---------------------------------|---------------------------------|---------------------------------|---------------------------------|---------------------------------|---------------------------------|---------------------------------|
| Methods   | R@1                             | R@2                             | R@3                             | AvgR                            | R@1                             | R@2                             | R@3                             | AvgR                            |
| GT        | 64.36                           | 88.75                           | 95.56                           | 1.74                            | 20.83                           | 33.66                           | 40.47                           | 33.13                           |
| 10%       | 2.57                            | 3.75                            | 5.03                            | 259.73                          | 3.75                            | 5.33                            | 7.40                            | 213.17                          |
| 50%       | 8.00                            | 14.71                           | 18.36                           | 104.85                          | 13.03                           | 20.73                           | 25.07                           | 77.08                           |
| 100%      | 14.51                           | 21.72                           | 28.73                           | 56.63                           | 22.41                           | 34.45                           | 40.57                           | 31.42                           |

trials with obvious answers, and filter out workers who fail them. In each example, two videos are presented side-by-side, randomly swapped for each example. Participants then compare the videos for the given edit text descriptions, and select which of the two follows the instructions better. TMED was chosen over MDM-BP in 65.8% of the comparisons, which again confirms the previous conclusions.

## D GPT-4 BASED ANNOTATION IN MDM-BP

For the MDM-BP baseline described in Section 5.2 of the main paper, in order to automatically extract which body parts to edit, we prompt GPT-4 through its publicly available API: https://openai.com/api/.

Our approach is similar to the one followed in [Athanasiou et al. 2023]. We feed the edit text to the LLM and prompt it to provide the body parts that should be edited in order to achieve the given motion edit. We use the same body parts as in [Athanasiou et al. 2023] and give them to GPT-4 as the list to choose from. We experimented with various prompts and eventually used the following: You will be given an edit text that is supposed to be used to edit a motion. Your task is given the text to determine what are the parts of the motion that should change based on that edit text. The instructions for this task are to choose your answers from the list below: left arm, right arm, left leg, buttocks, waist, right leg, torso, neck Here are some examples of the question and answer pairs for this task: 'Question: What are the body parts that should be edited in the motion if the edit text is: faster?' 'Answer: right leg, left leg, buttocks, left arm, right arm, torso, neck' 'Question: What are the body parts that should be edited in the motion if the edit text is: do it with the opposite leg?' 'Answer: right leg, left leg, buttocks' ... 'Question: What are the body parts that should be edited in the motion if the edit text is: stop moving in the end?' 'Answer: right leg, left leg, buttocks, left arm, right arm, torso,

14 · Nikos Athanasiou, Alpár Cseke, Markos Diomataris, Michael J. Black, and Gül Varol

| Edit Text                                                               | Body Parts to Edit                                              |
|-------------------------------------------------------------------------|-----------------------------------------------------------------|
| spread your legs more as you jump                                       | right leg, left leg                                             |
| Instead of using both hands use one hand                                | right arm, left arm                                             |
| instead of completing throw- ing motion bring hands close as if to pray | right arm, left arm                                             |
| don't swing from side to side                                           | left leg, right leg, buttocks, torso                            |
| turn around                                                             | right leg, left leg, buttocks, right arm, left arm, torso, neck |
| Sit little bit slower                                                   | right leg, left leg, buttocks                                   |
| bend knees more start swing- ing left                                   | right leg, left leg, buttocks                                   |
| move faster in the end add back tilt                                    | right leg, left leg, buttocks, left arm, right arm, torso, neck |
| step forward sooner and reach straight ahead                            | right leg, left leg, buttocks, torso                            |
| oppose arms that started movement                                       | right arm, left arm                                             |
| reach further back and lower with the same hand                         | right arm, torso                                                |
| raise your hand faster and a little higher                              | right arm, left arm                                             |
| jump slower and only four times                                         | right leg, left leg, buttocks                                   |

Table A.4. Example GPT responses: We show several input-output pairs for the MDM-BP baseline that uses automatically extracted body parts from edit text. GPT reasonably identifies the body parts that need to be edited e.g., 'turn around' involves the whole body, 'raise your hand faster and a little higher' involves both arms as this cannot be inferred purely from text.

neck' 'Question: What are the body parts that should be edited in the motion if the edit text is: [EDIT TEXT]'

We qualitatively observe that by giving some examples in the prompt, GPT-4 is able to provide better responses. In Table A.4 we present examples of edit texts and the respective GPT-4 responses.
Nikos Athanasiou Alpár Cseke Markos Diomataris Michael J. Black Gül Varol
MotionFix: Text-Driven 3D Human Motion Editing
The focus of this paper is on 3D motion editing. Given a 3D human motion and
a textual description of the desired modification, our goal is to generate an
edited motion as described by the text. The key challenges include the scarcity
of training data and the need to design a model that accurately edits the
source motion. In this paper, we address both challenges. We propose a
methodology to semi-automatically collect a dataset of triplets comprising (i)
a source motion, (ii) a target motion, and (iii) an edit text, introducing the
new MotionFix dataset. Access to this data allows us to train a conditional
diffusion model, TMED, that takes both the source motion and the edit text as
input. We develop several baselines to evaluate our model, comparing it against
models trained solely on text-motion pair datasets, and demonstrate the
superior performance of our model trained on triplets. We also introduce new
retrieval-based metrics for motion editing, establishing a benchmark on the
evaluation set of MotionFix. Our results are promising, paving the way for
further research in fine-grained motion generation. Code, models, and data are
available at https://motionfix.is.tue.mpg.de/ .