title: DreamVideo: Composing Your Dream Videos with Customized Subject and
    Motion
summary: Customized generation using diffusion models has made impressive progress in
  image generation, but remains unsatisfactory in the challenging video
  generation task, as it requires the controllability of both subjects and
  motions. To that end, we present DreamVideo, a novel approach to generating
  personalized videos from a few static images of the desired subject and a few
  videos of target motion. DreamVideo decouples this task into two stages,
  subject learning and motion learning, by leveraging a pre-trained video
  diffusion model. The subject learning aims to accurately capture the fine
  appearance of the subject from provided images, which is achieved by combining
  textual inversion and fine-tuning of our carefully designed identity adapter.
  In motion learning, we architect a motion adapter and fine-tune it on the given
  videos to effectively model the target motion pattern. Combining these two
  lightweight and efficient adapters allows for flexible customization of any
  subject with any motion. Extensive experimental results demonstrate the
  superior performance of our DreamVideo over the state-of-the-art methods for
  customized video generation. Our project page is at
  https://dreamvideo-t2v.github.io.
paper_authors._id: 65728bb22f6f6e71697af3ab
paper_authors.name: Yujie Wei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-11T07:44:43.880Z
paper_authors._id: 637f70d6fab5db9101c3dfc8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637f70d6fab5db9101c3dfc8/NgkYNXWLDavLbrnCby2Fl.jpeg
paper_authors.fullname: Yujie Wei
paper_authors.type: user
paper_authors.user: weilllllls
paper_authors._id: 65728bb22f6f6e71697af3ac
paper_authors.name: Shiwei Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:53:17.938Z
paper_authors._id: 63a1200045edac9f7508bae9
paper_authors.avatarUrl: /avatars/e84f0b045f32c5b8b4da43458650b925.svg
paper_authors.fullname: Shiwei Zhang
paper_authors.type: user
paper_authors.user: StevenZhang
paper_authors._id: 65728bb22f6f6e71697af3ad
paper_authors.name: Zhiwu Qing
paper_authors._id: 65728bb22f6f6e71697af3ae
paper_authors.name: Hangjie Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:53:33.583Z
paper_authors._id: 649d54b314afbb10ce2a9eeb
paper_authors.avatarUrl: /avatars/15c325d8c2273ff63569f23015e98486.svg
paper_authors.fullname: Hangjie Yuan
paper_authors.type: user
paper_authors.user: JacobYuan
paper_authors._id: 65728bb22f6f6e71697af3af
paper_authors.name: Zhiheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:54:03.819Z
paper_authors._id: 6479925ab77e18dbf640bd67
paper_authors.avatarUrl: /avatars/bb52ecd22ca4b49157f8668be35409e7.svg
paper_authors.fullname: Zhiheng Liu
paper_authors.type: user
paper_authors.user: Johanan0528
paper_authors._id: 65728bb22f6f6e71697af3b0
paper_authors.name: Yu Liu
paper_authors._id: 65728bb22f6f6e71697af3b1
paper_authors.name: Yingya Zhang
paper_authors._id: 65728bb22f6f6e71697af3b2
paper_authors.name: Jingren Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-08T10:55:04.255Z
paper_authors._id: 602f88f5e8149a962412a667
paper_authors.avatarUrl: /avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Jingren
paper_authors._id: 65728bb22f6f6e71697af3b3
paper_authors.name: Hongming Shan
paper_title: DreamVideo: Composing Your Dream Videos with Customized Subject and
    Motion
paper_summary: Customized generation using diffusion models has made impressive progress in
  image generation, but remains unsatisfactory in the challenging video
  generation task, as it requires the controllability of both subjects and
  motions. To that end, we present DreamVideo, a novel approach to generating
  personalized videos from a few static images of the desired subject and a few
  videos of target motion. DreamVideo decouples this task into two stages,
  subject learning and motion learning, by leveraging a pre-trained video
  diffusion model. The subject learning aims to accurately capture the fine
  appearance of the subject from provided images, which is achieved by combining
  textual inversion and fine-tuning of our carefully designed identity adapter.
  In motion learning, we architect a motion adapter and fine-tune it on the given
  videos to effectively model the target motion pattern. Combining these two
  lightweight and efficient adapters allows for flexible customization of any
  subject with any motion. Extensive experimental results demonstrate the
  superior performance of our DreamVideo over the state-of-the-art methods for
  customized video generation. Our project page is at
  https://dreamvideo-t2v.github.io.
paper_ai_summary: DreamVideo uses a pre-trained video diffusion model and two lightweight adapters to generate customized videos by learning subjects from images and motions from videos, outperforming existing methods.
paper_ai_keywords: diffusion model subject learning motion learning textual inversion identity adapter motion adapter