title: Customizing Motion in Text-to-Video Diffusion Models
summary: We introduce an approach for augmenting text-to-video generation models with
  customized motions, extending their capabilities beyond the motions depicted in
  the original training data. By leveraging a few video samples demonstrating
  specific movements as input, our method learns and generalizes the input motion
  patterns for diverse, text-specified scenarios. Our contributions are
  threefold. First, to achieve our results, we finetune an existing text-to-video
  model to learn a novel mapping between the depicted motion in the input
  examples to a new unique token. To avoid overfitting to the new custom motion,
  we introduce an approach for regularization over videos. Second, by leveraging
  the motion priors in a pretrained model, our method can produce novel videos
  featuring multiple people doing the custom motion, and can invoke the motion in
  combination with other motions. Furthermore, our approach extends to the
  multimodal customization of motion and appearance of individualized subjects,
  enabling the generation of videos featuring unique characters and distinct
  motions. Third, to validate our method, we introduce an approach for
  quantitatively evaluating the learned custom motion and perform a systematic
  ablation study. We show that our method significantly outperforms prior
  appearance-based customization approaches when extended to the motion
  customization task.
paper_authors._id: 65767e816da136b50fdcea2c
paper_authors.name: Joanna Materzynska
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-11T08:27:22.658Z
paper_authors._id: 60ef074b89938e400b470da8
paper_authors.avatarUrl: /avatars/c914070df46fb2dce9067d0eafc88238.svg
paper_authors.fullname: Joanna Materzynska
paper_authors.type: user
paper_authors.user: jomat
paper_authors._id: 65767e816da136b50fdcea2d
paper_authors.name: Josef Sivic
paper_authors._id: 65767e816da136b50fdcea2e
paper_authors.name: Eli Shechtman
paper_authors._id: 65767e816da136b50fdcea2f
paper_authors.name: Antonio Torralba
paper_authors._id: 65767e816da136b50fdcea30
paper_authors.name: Richard Zhang
paper_authors._id: 65767e816da136b50fdcea31
paper_authors.name: Bryan Russell
paper_title: Customizing Motion in Text-to-Video Diffusion Models
paper_summary: We introduce an approach for augmenting text-to-video generation models with
  customized motions, extending their capabilities beyond the motions depicted in
  the original training data. By leveraging a few video samples demonstrating
  specific movements as input, our method learns and generalizes the input motion
  patterns for diverse, text-specified scenarios. Our contributions are
  threefold. First, to achieve our results, we finetune an existing text-to-video
  model to learn a novel mapping between the depicted motion in the input
  examples to a new unique token. To avoid overfitting to the new custom motion,
  we introduce an approach for regularization over videos. Second, by leveraging
  the motion priors in a pretrained model, our method can produce novel videos
  featuring multiple people doing the custom motion, and can invoke the motion in
  combination with other motions. Furthermore, our approach extends to the
  multimodal customization of motion and appearance of individualized subjects,
  enabling the generation of videos featuring unique characters and distinct
  motions. Third, to validate our method, we introduce an approach for
  quantitatively evaluating the learned custom motion and perform a systematic
  ablation study. We show that our method significantly outperforms prior
  appearance-based customization approaches when extended to the motion
  customization task.
paper_ai_keywords: text-to-video generation models customized motions motion patterns finetuning novel mapping unique token regularization over videos motion priors pretrained model multimodal customization motion customization task appearance-based customization approaches