DreamVideo: Composing Your Dream Videos with Customized Subject and
  Motion
Customized generation using diffusion models has made impressive progress in
image generation, but remains unsatisfactory in the challenging video
generation task, as it requires the controllability of both subjects and
motions. To that end, we present DreamVideo, a novel approach to generating
personalized videos from a few static images of the desired subject and a few
videos of target motion. DreamVideo decouples this task into two stages,
subject learning and motion learning, by leveraging a pre-trained video
diffusion model. The subject learning aims to accurately capture the fine
appearance of the subject from provided images, which is achieved by combining
textual inversion and fine-tuning of our carefully designed identity adapter.
In motion learning, we architect a motion adapter and fine-tune it on the given
videos to effectively model the target motion pattern. Combining these two
lightweight and efficient adapters allows for flexible customization of any
subject with any motion. Extensive experimental results demonstrate the
superior performance of our DreamVideo over the state-of-the-art methods for
customized video generation. Our project page is at
https://dreamvideo-t2v.github.io.
65728bb22f6f6e71697af3ab
Yujie Wei
claimed_verified
2024-04-11T07:44:43.880Z
637f70d6fab5db9101c3dfc8
https://cdn-avatars.huggingface.co/v1/production/uploads/637f70d6fab5db9101c3dfc8/NgkYNXWLDavLbrnCby2Fl.jpeg
Yujie Wei
user
weilllllls
65728bb22f6f6e71697af3ac
Shiwei Zhang
admin_assigned
2023-12-08T10:53:17.938Z
63a1200045edac9f7508bae9
/avatars/e84f0b045f32c5b8b4da43458650b925.svg
Shiwei Zhang
user
StevenZhang
65728bb22f6f6e71697af3ad
Zhiwu Qing
65728bb22f6f6e71697af3ae
Hangjie Yuan
admin_assigned
2023-12-08T10:53:33.583Z
649d54b314afbb10ce2a9eeb
/avatars/15c325d8c2273ff63569f23015e98486.svg
Hangjie Yuan
user
JacobYuan
65728bb22f6f6e71697af3af
Zhiheng Liu
admin_assigned
2023-12-08T10:54:03.819Z
6479925ab77e18dbf640bd67
/avatars/bb52ecd22ca4b49157f8668be35409e7.svg
Zhiheng Liu
user
Johanan0528
65728bb22f6f6e71697af3b0
Yu Liu
65728bb22f6f6e71697af3b1
Yingya Zhang
65728bb22f6f6e71697af3b2
Jingren Zhou
admin_assigned
2023-12-08T10:55:04.255Z
602f88f5e8149a962412a667
/avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
Zhou
user
Jingren
65728bb22f6f6e71697af3b3
Hongming Shan
DreamVideo: Composing Your Dream Videos with Customized Subject and
  Motion
Customized generation using diffusion models has made impressive progress in
image generation, but remains unsatisfactory in the challenging video
generation task, as it requires the controllability of both subjects and
motions. To that end, we present DreamVideo, a novel approach to generating
personalized videos from a few static images of the desired subject and a few
videos of target motion. DreamVideo decouples this task into two stages,
subject learning and motion learning, by leveraging a pre-trained video
diffusion model. The subject learning aims to accurately capture the fine
appearance of the subject from provided images, which is achieved by combining
textual inversion and fine-tuning of our carefully designed identity adapter.
In motion learning, we architect a motion adapter and fine-tune it on the given
videos to effectively model the target motion pattern. Combining these two
lightweight and efficient adapters allows for flexible customization of any
subject with any motion. Extensive experimental results demonstrate the
superior performance of our DreamVideo over the state-of-the-art methods for
customized video generation. Our project page is at
https://dreamvideo-t2v.github.io.
DreamVideo uses a pre-trained video diffusion model and two lightweight adapters to generate customized videos by learning subjects from images and motions from videos, outperforming existing methods.
diffusion model subject learning motion learning textual inversion identity adapter motion adapter