title: VMC: Video Motion Customization using Temporal Attention Adaption for
    Text-to-Video Diffusion Models
summary: Text-to-video diffusion models have advanced video generation significantly.
  However, customizing these models to generate videos with tailored motions
  presents a substantial challenge. In specific, they encounter hurdles in (a)
  accurately reproducing motion from a target video, and (b) creating diverse
  visual variations. For example, straightforward extensions of static image
  customization methods to video often lead to intricate entanglements of
  appearance and motion data. To tackle this, here we present the Video Motion
  Customization (VMC) framework, a novel one-shot tuning approach crafted to
  adapt temporal attention layers within video diffusion models. Our approach
  introduces a novel motion distillation objective using residual vectors between
  consecutive frames as a motion reference. The diffusion process then preserves
  low-frequency motion trajectories while mitigating high-frequency
  motion-unrelated noise in image space. We validate our method against
  state-of-the-art video generative models across diverse real-world motions and
  contexts. Our codes, data and the project demo can be found at
  https://video-motion-customization.github.io
paper_authors._id: 656e90d4e9d3e5f3f7babfc6
paper_authors.name: Hyeonho Jeong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T09:08:50.420Z
paper_authors._id: 656ee8008bb9f4f8d95bd8f7
paper_authors.avatarUrl: /avatars/4069d70f1279d928da521211c495d638.svg
paper_authors.fullname: Hyeonho Jeong
paper_authors.type: user
paper_authors.user: hyeonho-jeong-video
paper_authors._id: 656e90d4e9d3e5f3f7babfc7
paper_authors.name: Geon Yeong Park
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T09:08:56.977Z
paper_authors._id: 64cfff0abf39f9c8bebd2a6f
paper_authors.avatarUrl: /avatars/e3405136c8af5065c287eff718c78b5b.svg
paper_authors.fullname: Geon Yeong Park
paper_authors.type: user
paper_authors.user: geonyoung-park
paper_authors._id: 656e90d4e9d3e5f3f7babfc8
paper_authors.name: Jong Chul Ye
paper_title: VMC: Video Motion Customization using Temporal Attention Adaption for
    Text-to-Video Diffusion Models
paper_summary: Text-to-video diffusion models have advanced video generation significantly.
  However, customizing these models to generate videos with tailored motions
  presents a substantial challenge. In specific, they encounter hurdles in (a)
  accurately reproducing motion from a target video, and (b) creating diverse
  visual variations. For example, straightforward extensions of static image
  customization methods to video often lead to intricate entanglements of
  appearance and motion data. To tackle this, here we present the Video Motion
  Customization (VMC) framework, a novel one-shot tuning approach crafted to
  adapt temporal attention layers within video diffusion models. Our approach
  introduces a novel motion distillation objective using residual vectors between
  consecutive frames as a motion reference. The diffusion process then preserves
  low-frequency motion trajectories while mitigating high-frequency
  motion-unrelated noise in image space. We validate our method against
  state-of-the-art video generative models across diverse real-world motions and
  contexts. Our codes, data and the project demo can be found at
  https://video-motion-customization.github.io
paper_ai_summary: The VMC framework enhances video generation by tuning temporal attention layers in diffusion models to accurately reproduce and diversify motion with a motion distillation objective.
paper_ai_keywords: text-to-video diffusion models video diffusion models temporal attention layers motion customization motion distillation residual vectors low-frequency motion trajectories high-frequency motion-unrelated noise state-of-the-art video generative models