MotionDirector: Motion Customization of Text-to-Video Diffusion Models
Large-scale pre-trained diffusion models have exhibited remarkable
capabilities in diverse video generations. Given a set of video clips of the
same motion concept, the task of Motion Customization is to adapt existing
text-to-video diffusion models to generate videos with this motion. For
example, generating a video with a car moving in a prescribed manner under
specific camera movements to make a movie, or a video illustrating how a bear
would lift weights to inspire creators. Adaptation methods have been developed
for customizing appearance like subject or style, yet unexplored for motion. It
is straightforward to extend mainstream adaption methods for motion
customization, including full model tuning, parameter-efficient tuning of
additional layers, and Low-Rank Adaptions (LoRAs). However, the motion concept
learned by these methods is often coupled with the limited appearances in the
training videos, making it difficult to generalize the customized motion to
other appearances. To overcome this challenge, we propose MotionDirector, with
a dual-path LoRAs architecture to decouple the learning of appearance and
motion. Further, we design a novel appearance-debiased temporal loss to
mitigate the influence of appearance on the temporal training objective.
Experimental results show the proposed method can generate videos of diverse
appearances for the customized motions. Our method also supports various
downstream applications, such as the mixing of different videos with their
appearance and motion respectively, and animating a single image with
customized motions. Our code and model weights will be released.
65294fd7b20d500447d25548
Rui Zhao
claimed_verified
2023-10-16T07:22:19.915Z
652b83b73b5997ed71a310f2
https://cdn-avatars.huggingface.co/v1/production/uploads/652b83b73b5997ed71a310f2/ipCpdeHUp4-0OmRz5z8IW.png
Rui Zhao
user
ruizhaocv
65294fd7b20d500447d25549
Yuchao Gu
admin_assigned
2023-10-13T15:54:53.570Z
63021630a35b21bd8a53305a
/avatars/7a7e8b39749eda61e57d8a1908726558.svg
Gu Yuchao
user
guyuchao
65294fd7b20d500447d2554a
Jay Zhangjie Wu
admin_assigned
2023-12-06T11:32:15.991Z
633aaf695df91da9cea92960
https://cdn-avatars.huggingface.co/v1/production/uploads/633aaf695df91da9cea92960/h1oOWUudbdlsd63Q5w0hM.png
Jay Wu
user
jayw
65294fd7b20d500447d2554b
David Junhao Zhang
admin_assigned
2023-10-13T15:55:47.084Z
6468855763a564ba347e1efa
/avatars/caa99c3318c1d6d21be44d2d32795a62.svg
David Junhao ZHANG
user
Junhao233
65294fd7b20d500447d2554c
Jiawei Liu
admin_assigned
2023-10-13T15:56:10.893Z
64230048a73327caad9d8241
/avatars/ac115f1a21c743db7c925f0e18451145.svg
Jiawei Liu
user
JiaweiLIU
65294fd7b20d500447d2554d
Weijia Wu
admin_assigned
2023-10-13T15:56:17.836Z
6345a93afe134dfd7a0cfabd
/avatars/65130ce06b1c72ab1066678419731d88.svg
wu weijia
user
weijiawu
65294fd7b20d500447d2554e
Jussi Keppo
65294fd7b20d500447d2554f
Mike Zheng Shou
admin_assigned
2023-10-13T15:56:40.003Z
63a55320ce5763e06f78519c
https://cdn-avatars.huggingface.co/v1/production/uploads/1671779060549-noauth.jpeg
Mike Shou
user
mikeshou
MotionDirector: Motion Customization of Text-to-Video Diffusion Models
Large-scale pre-trained diffusion models have exhibited remarkable
capabilities in diverse video generations. Given a set of video clips of the
same motion concept, the task of Motion Customization is to adapt existing
text-to-video diffusion models to generate videos with this motion. For
example, generating a video with a car moving in a prescribed manner under
specific camera movements to make a movie, or a video illustrating how a bear
would lift weights to inspire creators. Adaptation methods have been developed
for customizing appearance like subject or style, yet unexplored for motion. It
is straightforward to extend mainstream adaption methods for motion
customization, including full model tuning, parameter-efficient tuning of
additional layers, and Low-Rank Adaptions (LoRAs). However, the motion concept
learned by these methods is often coupled with the limited appearances in the
training videos, making it difficult to generalize the customized motion to
other appearances. To overcome this challenge, we propose MotionDirector, with
a dual-path LoRAs architecture to decouple the learning of appearance and
motion. Further, we design a novel appearance-debiased temporal loss to
mitigate the influence of appearance on the temporal training objective.
Experimental results show the proposed method can generate videos of diverse
appearances for the customized motions. Our method also supports various
downstream applications, such as the mixing of different videos with their
appearance and motion respectively, and animating a single image with
customized motions. Our code and model weights will be released.
MotionDirector decouples motion adaptation from appearance in text-to-video diffusion models using a dual-path LoRAs architecture and appearance-debiased temporal loss.
diffusion models Motion Customization full model tuning parameter-efficient tuning Low-Rank Adaptions (LoRAs) appearance-debiased temporal loss MotionDirector dual-path LoRAs architecture downstream applications