title: MotionDirector: Motion Customization of Text-to-Video Diffusion Models
summary: Large-scale pre-trained diffusion models have exhibited remarkable
  capabilities in diverse video generations. Given a set of video clips of the
  same motion concept, the task of Motion Customization is to adapt existing
  text-to-video diffusion models to generate videos with this motion. For
  example, generating a video with a car moving in a prescribed manner under
  specific camera movements to make a movie, or a video illustrating how a bear
  would lift weights to inspire creators. Adaptation methods have been developed
  for customizing appearance like subject or style, yet unexplored for motion. It
  is straightforward to extend mainstream adaption methods for motion
  customization, including full model tuning, parameter-efficient tuning of
  additional layers, and Low-Rank Adaptions (LoRAs). However, the motion concept
  learned by these methods is often coupled with the limited appearances in the
  training videos, making it difficult to generalize the customized motion to
  other appearances. To overcome this challenge, we propose MotionDirector, with
  a dual-path LoRAs architecture to decouple the learning of appearance and
  motion. Further, we design a novel appearance-debiased temporal loss to
  mitigate the influence of appearance on the temporal training objective.
  Experimental results show the proposed method can generate videos of diverse
  appearances for the customized motions. Our method also supports various
  downstream applications, such as the mixing of different videos with their
  appearance and motion respectively, and animating a single image with
  customized motions. Our code and model weights will be released.
paper_authors._id: 65294fd7b20d500447d25548
paper_authors.name: Rui Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-16T07:22:19.915Z
paper_authors._id: 652b83b73b5997ed71a310f2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/652b83b73b5997ed71a310f2/ipCpdeHUp4-0OmRz5z8IW.png
paper_authors.fullname: Rui Zhao
paper_authors.type: user
paper_authors.user: ruizhaocv
paper_authors._id: 65294fd7b20d500447d25549
paper_authors.name: Yuchao Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-13T15:54:53.570Z
paper_authors._id: 63021630a35b21bd8a53305a
paper_authors.avatarUrl: /avatars/7a7e8b39749eda61e57d8a1908726558.svg
paper_authors.fullname: Gu Yuchao
paper_authors.type: user
paper_authors.user: guyuchao
paper_authors._id: 65294fd7b20d500447d2554a
paper_authors.name: Jay Zhangjie Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T11:32:15.991Z
paper_authors._id: 633aaf695df91da9cea92960
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/633aaf695df91da9cea92960/9T4y1ru5wt5iKUUqf9_Tt.png
paper_authors.fullname: Jay Wu
paper_authors.type: user
paper_authors.user: jayw
paper_authors._id: 65294fd7b20d500447d2554b
paper_authors.name: David Junhao Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-13T15:55:47.084Z
paper_authors._id: 6468855763a564ba347e1efa
paper_authors.avatarUrl: /avatars/caa99c3318c1d6d21be44d2d32795a62.svg
paper_authors.fullname: David Junhao ZHANG
paper_authors.type: user
paper_authors.user: Junhao233
paper_authors._id: 65294fd7b20d500447d2554c
paper_authors.name: Jiawei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-13T15:56:10.893Z
paper_authors._id: 64230048a73327caad9d8241
paper_authors.avatarUrl: /avatars/ac115f1a21c743db7c925f0e18451145.svg
paper_authors.fullname: Jiawei Liu
paper_authors.type: user
paper_authors.user: JiaweiLIU
paper_authors._id: 65294fd7b20d500447d2554d
paper_authors.name: Weijia Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-13T15:56:17.836Z
paper_authors._id: 6345a93afe134dfd7a0cfabd
paper_authors.avatarUrl: /avatars/65130ce06b1c72ab1066678419731d88.svg
paper_authors.fullname: wu weijia
paper_authors.type: user
paper_authors.user: weijiawu
paper_authors._id: 65294fd7b20d500447d2554e
paper_authors.name: Jussi Keppo
paper_authors._id: 65294fd7b20d500447d2554f
paper_authors.name: Mike Zheng Shou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-13T15:56:40.003Z
paper_authors._id: 63a55320ce5763e06f78519c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1671779060549-noauth.jpeg
paper_authors.fullname: Mike Shou
paper_authors.type: user
paper_authors.user: mikeshou
paper_title: MotionDirector: Motion Customization of Text-to-Video Diffusion Models
paper_summary: Large-scale pre-trained diffusion models have exhibited remarkable
  capabilities in diverse video generations. Given a set of video clips of the
  same motion concept, the task of Motion Customization is to adapt existing
  text-to-video diffusion models to generate videos with this motion. For
  example, generating a video with a car moving in a prescribed manner under
  specific camera movements to make a movie, or a video illustrating how a bear
  would lift weights to inspire creators. Adaptation methods have been developed
  for customizing appearance like subject or style, yet unexplored for motion. It
  is straightforward to extend mainstream adaption methods for motion
  customization, including full model tuning, parameter-efficient tuning of
  additional layers, and Low-Rank Adaptions (LoRAs). However, the motion concept
  learned by these methods is often coupled with the limited appearances in the
  training videos, making it difficult to generalize the customized motion to
  other appearances. To overcome this challenge, we propose MotionDirector, with
  a dual-path LoRAs architecture to decouple the learning of appearance and
  motion. Further, we design a novel appearance-debiased temporal loss to
  mitigate the influence of appearance on the temporal training objective.
  Experimental results show the proposed method can generate videos of diverse
  appearances for the customized motions. Our method also supports various
  downstream applications, such as the mixing of different videos with their
  appearance and motion respectively, and animating a single image with
  customized motions. Our code and model weights will be released.
paper_ai_keywords: diffusion models Motion Customization text-to-video diffusion models full model tuning parameter-efficient tuning Low-Rank Adaptations (LoRAs) MotionDirector dual-path LoRAs architecture appearance-debiased temporal loss downstream applications mixing of different videos animating a single image