title: Video Motion Transfer with Diffusion Transformers
summary: We propose DiTFlow, a method for transferring the motion of a reference video
  to a newly synthesized one, designed specifically for Diffusion Transformers
  (DiT). We first process the reference video with a pre-trained DiT to analyze
  cross-frame attention maps and extract a patch-wise motion signal called the
  Attention Motion Flow (AMF). We guide the latent denoising process in an
  optimization-based, training-free, manner by optimizing latents with our AMF
  loss to generate videos reproducing the motion of the reference one. We also
  apply our optimization strategy to transformer positional embeddings, granting
  us a boost in zero-shot motion transfer capabilities. We evaluate DiTFlow
  against recently published methods, outperforming all across multiple metrics
  and human evaluation.
paper_authors._id: 67592b708ac3b988434f8261
paper_authors.name: Alexander Pondaven
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-11T09:19:44.410Z
paper_authors._id: 63544cacb8aa3b0b3ebd9800
paper_authors.avatarUrl: /avatars/7868b79742de27074376d5698e281003.svg
paper_authors.fullname: Alexander Pondaven
paper_authors.type: user
paper_authors.user: alexpondaven
paper_authors._id: 67592b708ac3b988434f8262
paper_authors.name: Aliaksandr Siarohin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T10:35:51.963Z
paper_authors._id: 64276311eb9a0ed86180715b
paper_authors.avatarUrl: /avatars/76f933cd549f10e5e2db379de235d304.svg
paper_authors.fullname: Aliaksandr Siarohin
paper_authors.type: user
paper_authors.user: aliaksandr-siarohin
paper_authors._id: 67592b708ac3b988434f8263
paper_authors.name: Sergey Tulyakov
paper_authors._id: 67592b708ac3b988434f8264
paper_authors.name: Philip Torr
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T10:36:23.031Z
paper_authors._id: 6565ed28a5ec0231cb07225f
paper_authors.avatarUrl: /avatars/7f95bba9aa7811d56eecb380827abfac.svg
paper_authors.fullname: prof philip torr
paper_authors.type: user
paper_authors.user: philiptorr
paper_authors._id: 67592b708ac3b988434f8265
paper_authors.name: Fabio Pizzati
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-11T10:36:52.234Z
paper_authors._id: 62bdcba92dd4214c0622ea30
paper_authors.avatarUrl: /avatars/b62d86d01597473e05f9991ec04baf57.svg
paper_authors.fullname: Fabio Pizzati
paper_authors.type: user
paper_authors.user: fabvio
paper_title: Video Motion Transfer with Diffusion Transformers
paper_summary: We propose DiTFlow, a method for transferring the motion of a reference video
  to a newly synthesized one, designed specifically for Diffusion Transformers
  (DiT). We first process the reference video with a pre-trained DiT to analyze
  cross-frame attention maps and extract a patch-wise motion signal called the
  Attention Motion Flow (AMF). We guide the latent denoising process in an
  optimization-based, training-free, manner by optimizing latents with our AMF
  loss to generate videos reproducing the motion of the reference one. We also
  apply our optimization strategy to transformer positional embeddings, granting
  us a boost in zero-shot motion transfer capabilities. We evaluate DiTFlow
  against recently published methods, outperforming all across multiple metrics
  and human evaluation.
paper_ai_keywords: Diffusion Transformers (DiT) Attention Motion Flow (AMF) latent denoising positional embeddings zero-shot motion transfer