MotionFix: Text-Driven 3D Human Motion Editing

We use a sequence of SMPL [Loper et al. 2015] body parameters to represent a human motion. SMPL is a linear function that maps the shape, and pose parameters of J joints, along with the global body translation and orientation, to a 3D mesh. The joint positions, J_p, can be obtained from vertices via the learned SMPL joint regressor. Following previous work [Petrovich et al. 2022] that discards the shape parameters, we set the shape parameters to zero (mean shape), since motion is parameterized primarily by pose parameters.

Various alternative representations have been used based on joint positions with respect to the local coordinate system of the body [Guo et al. 2022a; Holden et al. 2016; Starke et al. 2019]. Unlike prior works [Guo et al. 2022a; Tevet et al. 2023; Zhang et al. 2023a] that fit SMPL bodies to skeleton generations, we aim to enable direct regression of SMPL parameters, bypassing the need of a costly post-processing optimization [Bogo et al. 2016], and thus making our method ready to use for animation frameworks.

A common approach for representing SMPL pose parameters within a learning framework is to employ 6D rotations [Zhou et al. 2019], and to apply first-frame canonicalization for motions [Athanasiou et al. 2022; Petrovich et al. 2022]. Similarly, we canonicalize our motions prior to training, so that all face the same direction in the first frame and have the same initial global position. Inspired by [Holden et al. 2016; Petrovich et al. 2024], we represent the global body translation as differences between consecutive frames. Supervising with such relative translations helps the denoiser to generate better trajectories, as we observed unsmooth generations when using the absolute translation. Similar to STMC [Petrovich et al. 2024], we factor out the z-rotation from the pelvis orientation and separately represent the global orientation as the xy-orientation and the z-orientation as the differences between rotations in consecutive frames (resulting in 12 features, i.e., 6D representation for xy and z). We represent the body pose with 6D rotations [Zhou et al. 2019]. Similar to [Petrovich et al. 2022], we exclude the hand joints as they mostly do not move in the datasets we use. We additionally append the local joint positions after removing z-rotation of the body [Holden et al. 2016; Petrovich et al. 2024] (resulting in 192 dimensional features with 6 × 21 for rotations and 22 × 3 for joints including the root joint). Thus, each motion frame has a dimension d_p = 207, consisting of 3 features for the global translation, 12 for the global orientation, and 192 for the body pose. The motion is represented as a sequence of the pose representations. During training, all features are normalized according to their mean and variance over the training set.

To learn TMED, we use our new training data, where each data sample comprises a source motion M_S, target motion M_T, and a language instruction L. We train a conditional diffusion model that learns to edit the source motion with respect to the instruction. We design a model similar to that of InstructPix2Pix [Brooks et al. 2023], where the generation from a random noise vector is conditioned on two further inputs L and M_S. Here, instead of a sequence of image patch tokens, the motion modality is represented as a variable-length sequence of motion frames. The noised target motion, the text condition L, and the source motion condition M_S are all fed as input to the denoiser at every diffusion step.

Diffusion models [Sohl-Dickstein et al. 2015] learn to gradually turn random noise into a sample from a data distribution by a sequence of denoising autoencoders. This is achieved by a diffusion process that adds noise ϵ_t to an input signal, M_T. We denote the noise level added to the input signal by using t, the diffusion timestep, as a superscript. This produces a diffused sample, \({M}^{t}_{T}\). The amount of noise added at timestep t = 1, …, N is defined a-priori through a noise schedule. We train a denoiser network, to reverse this process given the timestep t, the instruction L, the noised target motion \({M}^{t}_{T}\) and the source motion M_S. As supervision, the output of the denoiser network \(\tilde{M}^{t}_{T}\) is compared against the ground-truth denoised target motion M_T. Our model is therefore trained to minimize:

We use standard mean-squared-error as the loss function to compare the diffusion output with the ground-truth target motion. We choose to predict the denoised target motion, as we found this to produce better results visually than predicting the noise itself.

The architecture overview is illustrated in Figure 3 (left). Our model consists of multiple encoders for each input modality (E_T for timestep, E_L for text, and E_M for motion) and a transformer encoder D that operates on all inputs. The timestep t is encoded via E_T similar to MDM [Tevet et al. 2023], by first converting into a sinusoidal positional embedding, and then projecting through a feed-forward network (consisting of two linear layers with a SiLU activation [Elfwing et al. 2018] in between). As in [Tevet et al. 2023], we use the CLIP [Radford et al. 2021] text encoder for E_L. We pass the source and noised target motions through a linear layer (E_M), shared across frames, and obtain \(M^{enc}_{S}=E_M(M_S)\) and \({M^{t}_{T}}^{enc}=E_M({M}^{t}_{T})\). Given the variable duration of source and target motions, we add a learnable separation token SEP in between [Devlin et al. 2019] when appending them (so that the information on when the target motion ends and the source motion starts is communicated to the transformer). Once all encoded inputs have the same feature dimensionality d, they are combined into a single sequence to be fed to the transformer, as shown in Figure 3, and sinusoidal positional embeddings are subsequently added. During training, to enable classifier-free guidance, the source motion condition is randomly dropped 5% of the time, the text condition 5%, both conditions together 5%, and all the inputs are used 85% of the time. For sampling from a diffusion model with two conditions, we apply classifier-free guidance with respect to two conditions: the input motion M_S and the text instruction L. We introduce separate guidance scales \(s_{M_S}\) and s_L that allow adjusting the influence of each conditioning.

For simplicity, we now abuse the notation by dropping the timestep subscripts when deriving the sampling process. Our generative model, TMED, learns the probability distribution over the target motions, M_T, conditioned on the source motions and text condition, P(M_T∣M_S, L). Expanding this conditional probability gives:As in the original diffusion, we formulate this as a score function optimization problem by first taking the logarithm of Eq.(2):Then, the derivative with respect to the input of Eq.(3) gives the score estimate \(\tilde{e}_\theta (M_T, s_{M_S}, s_L)\), learned under classifier-free guidance:Hence, from Eq.(4), we sample from TMED using the modified score estimate of two-way conditioning a diffusion model as:We further ablate the guidance scales which control the generation at test time in Section 5.

Implementation details. All models are trained for 1000 epochs using cosine noise schedule with DDPM scheduler [Ho et al. 2020]. We use N = 300 diffusion timesteps, as we find this is a good compromise between speed and quality. The guidance scales are chosen for each model based on their best performance in the validation set of MotionFix  (\(S_L=2, s_{M_S}=2\)). We follow the same process for training MDM [Tevet et al. 2023] baselines described in the next section. In terms of architectural details, the dimensionality of the embeddings before inputting to the transformer is d = 512. We use a pre-trained and frozen CLIP [Radford et al. 2021] with all 77 token outputs of the ViT-B/32 backbone [Dosovitskiy et al. 2021] as our text encoder E_L. We use the text masks from E_L to mask the padded area of the text inputs. The motion encoder E_M that precedes the transformer is a simple linear projection with dimensionality d_p × d, where the feature dimension of each motion frame is d_p = 207 (as described in Section 4.1).

Similar to text-to-motion synthesis, distance-based metrics to evaluate motion generation quality is problematic due to multiple plausible ground-truth motions for a given text. Prior work has extensively used text-to-motion retrieval metrics for evaluating text-to-motion synthesis [Guo et al. 2022b; Tevet et al. 2023], by training a text-motion contrastive model and using its features. To evaluate motion editing, we introduce motion-to-motion retrieval metrics. Given a generated motion, we measure how well the source (generated-to-source retrieval) or the target motion (generated-to-target retrieval) can be retrieved. We use TMR [Petrovich et al. 2023] as the feature extractor, but train it ourselves to support our feature representation, using the same regime as in the original paper with HumanML3D data [Guo et al. 2022a]. We report standard metrics, R@1, R@2, R@3 and AvgR using a gallery size of 32 randomly sampled batches for retrieval from the test set. Recall at rank k (R@k) computes the percentage of times the correct motion is among the top k results. Note that we fix the batches so there is no randomness across evaluations. The performance is averaged across batches. We report results on the full test set as gallery in Tables A.2 and A.3 of the appendix, where the same conclusions hold. While the main performance measure is according to generated-to-target retrieval, we also monitor how close our generations remain to the source. As indicative values, we provide ground-truth (GT) values for the latter. We provide additional measures (FID, L2) and perceptual studies in Sections B and C of the appendix, respectively, which further confirm the results of our proposed retrieval-based benchmarking.

We report our main results in Table 2. We compare the performance of TMED trained on our MotionFix triplets against several baselines trained on the larger, HumanML3D text-motion pairs [Guo et al. 2022a]. We build our baselines by training MDM [Tevet et al. 2023] with our human motion representation and by repurposing this model for motion editing, described next.

We first introduce two simple baselines: (a) MDM that purely uses the edit text as input to the text-to-motion generation (i.e., without a source motion), and (b) MDM_S that additionally uses the source motion as input instead of noise during inference. For the latter, we also investigated reducing the number of diffusion steps when initializing from source; however, we observed performance drops and therefore kept the full 300 diffusion steps. Inspired by [Athanasiou et al. 2023], we design two additional strong baselines (MDM-BP_S and MDM-BP), that are based on body part labels extracted by querying GPT with the edit texts. We automatically detect body parts which are irrelevant to the text and keep them constant via masking. We again initialize the diffusion process either from the source motion (MDM-BP_S) or from noise (MDM-BP) for the body parts that need to change according to the GPT response. For more details on the query and example GPT outputs, we refer to Section D of the appendix.

We first observe from Table 2 that, for all the baselines, initializing from noise performs better than initializing from source motion. Our strong baselines based on body-part detection (MDM-BP, MDM-BP_S) clearly outperform the naive baselines. However, all baselines fall behind our TMED that successfully leverages the access to training triplets, and significantly outperforms alternatives.

Moreover, MDM-BP, MDM-BP_S are both strong baselines, but relying on GPT body part labels might not capture all edit types, such as the ones that require modifying the overall body. We demonstrate this further in our supplementary video from the project webpage and our qualitative comparisons (Section 5.4).

In the following, we investigate the effect of training data size and the guidance scales on the TMED model performance.

Training data size. In Table 3, we present the performance of TMED for different data sizes from MotionFix. We clearly observe, that increasing the data size has a large impact on the performance, justifying our data collection. The non-saturated trend is encouraging to scale up the training further.

Guidance hyperparameters. In Figure 4, we present how TMED performs across different guidance values for both conditions. x-axis controls the text guidance s_L, and y-axis controls the source motion guidance \(s_{M_S}\) at test time. We report both generated-to-target (left) and source-to-target (right) R@1 retrieval results. We observe that there needs to be a balance between the two guidance values, and that performances decrease towards the extremes (e.g., top left and bottom right corners of the plots, where only one of the two conditions have higher guidance). This highlights the need to rely on both conditions to perform the task.

We display several generations from TMED in Figure 5 to enable qualitative assessment. We observe that our model can perform different types of edits such as the addition of actions (“rotate wrists instead of stretching like yawn”), temporal edits in a motion (“get up a bit earlier”), speed edits (“slow down”) and combinations of these. We refer to our supplementary video for dynamic visualizations, which may be easier to interpret.

We additionally provide a qualitative comparison in Figure 7, between TMED and various baselines. We provide two comparisons for each baseline (top block for MDM_S, middle block for MDM-BP_S, and bottom block for MDM-B).

We observe that MDM_S picks up the action from the prompt, but fails to faithfully follow the source motion. In the first row, the generation by MDM_S raises both hands, instead of adjusting only the height of the hand raised in the source motion. Similarly, in the second row, MDM_S generation raises the arm but in front of the body and not higher as prompted by the edit text.

In the next two rows, we visualize MDM-BP_S results. Given the text “rotate wrists instead of stretching like yawn”, GPT correctly suggests editing both hands; however, the generated motion no longer resembles the source motion as the wide-open hands are not preserved. For the example edit “turn in the opposite direction”, all body parts are involved, but MDM-BP_S does not deviate too much from the source, perhaps because traditional text-to-motion generation models rarely see relative words such as “opposite direction”.

Finally, we illustrate generations from our strongest baseline MDM-BP in the last two rows. Both generations involve all parts of the body (e.g., “slow down”) making it hard to follow the source motion. In comparison, our model faithfully performs most of the edits. The disadvantage of TMED, on the other hand, might be the generalization to motion pairs where the TMR similarity is low as such edits were unseen during training. We briefly discuss more limitations in the following.