Video Motion Transfer with Diffusion Transformers
We propose DiTFlow, a method for transferring the motion of a reference video
to a newly synthesized one, designed specifically for Diffusion Transformers
(DiT). We first process the reference video with a pre-trained DiT to analyze
cross-frame attention maps and extract a patch-wise motion signal called the
Attention Motion Flow (AMF). We guide the latent denoising process in an
optimization-based, training-free, manner by optimizing latents with our AMF
loss to generate videos reproducing the motion of the reference one. We also
apply our optimization strategy to transformer positional embeddings, granting
us a boost in zero-shot motion transfer capabilities. We evaluate DiTFlow
against recently published methods, outperforming all across multiple metrics
and human evaluation.
67592b708ac3b988434f8261
Alexander Pondaven
claimed_verified
2024-12-11T09:19:44.410Z
63544cacb8aa3b0b3ebd9800
/avatars/7868b79742de27074376d5698e281003.svg
Alexander Pondaven
user
alexpondaven
67592b708ac3b988434f8262
Aliaksandr Siarohin
admin_assigned
2024-12-11T10:35:51.963Z
64276311eb9a0ed86180715b
/avatars/76f933cd549f10e5e2db379de235d304.svg
Aliaksandr Siarohin
user
aliaksandr-siarohin
67592b708ac3b988434f8263
Sergey Tulyakov
67592b708ac3b988434f8264
Philip Torr
admin_assigned
2024-12-11T10:36:23.031Z
6565ed28a5ec0231cb07225f
/avatars/7f95bba9aa7811d56eecb380827abfac.svg
prof philip torr
user
philiptorr
67592b708ac3b988434f8265
Fabio Pizzati
admin_assigned
2024-12-11T10:36:52.234Z
62bdcba92dd4214c0622ea30
/avatars/b62d86d01597473e05f9991ec04baf57.svg
Fabio Pizzati
user
fabvio
Video Motion Transfer with Diffusion Transformers
We propose DiTFlow, a method for transferring the motion of a reference video
to a newly synthesized one, designed specifically for Diffusion Transformers
(DiT). We first process the reference video with a pre-trained DiT to analyze
cross-frame attention maps and extract a patch-wise motion signal called the
Attention Motion Flow (AMF). We guide the latent denoising process in an
optimization-based, training-free, manner by optimizing latents with our AMF
loss to generate videos reproducing the motion of the reference one. We also
apply our optimization strategy to transformer positional embeddings, granting
us a boost in zero-shot motion transfer capabilities. We evaluate DiTFlow
against recently published methods, outperforming all across multiple metrics
and human evaluation.
DiTFlow uses Attention Motion Flow from a reference video to transfer motion to a synthesized video using Diffusion Transformers, outperforming other methods.
Diffusion Transformers DiT Attention Motion Flow AMF latent denoising process transformer positional embeddings zero-shot motion transfer