title: VideoMaker: Zero-shot Customized Video Generation with the Inherent
    Force of Video Diffusion Models
summary: Zero-shot customized video generation has gained significant attention due to
  its substantial application potential. Existing methods rely on additional
  models to extract and inject reference subject features, assuming that the
  Video Diffusion Model (VDM) alone is insufficient for zero-shot customized
  video generation. However, these methods often struggle to maintain consistent
  subject appearance due to suboptimal feature extraction and injection
  techniques. In this paper, we reveal that VDM inherently possesses the force to
  extract and inject subject features. Departing from previous heuristic
  approaches, we introduce a novel framework that leverages VDM's inherent force
  to enable high-quality zero-shot customized video generation. Specifically, for
  feature extraction, we directly input reference images into VDM and use its
  intrinsic feature extraction process, which not only provides fine-grained
  features but also significantly aligns with VDM's pre-trained knowledge. For
  feature injection, we devise an innovative bidirectional interaction between
  subject features and generated content through spatial self-attention within
  VDM, ensuring that VDM has better subject fidelity while maintaining the
  diversity of the generated video.Experiments on both customized human and
  object video generation validate the effectiveness of our framework.
paper_authors._id: 677229bf8103ad52cb7031b1
paper_authors.name: Tao Wu
paper_authors._id: 677229bf8103ad52cb7031b2
paper_authors.name: Yong Zhang
paper_authors._id: 677229bf8103ad52cb7031b3
paper_authors.name: Xiaodong Cun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-30T20:42:51.507Z
paper_authors._id: 63184c517ca1b876d99b7e0e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
paper_authors.fullname: Xiaodong Cun
paper_authors.type: user
paper_authors.user: vinthony
paper_authors._id: 677229bf8103ad52cb7031b4
paper_authors.name: Zhongang Qi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-30T20:42:45.157Z
paper_authors._id: 660cc64f1b14f691990c0ea0
paper_authors.avatarUrl: /avatars/f172d3120b22d745a41cc3f2eb499ce6.svg
paper_authors.fullname: Zhongang Qi
paper_authors.type: user
paper_authors.user: phoenixqza
paper_authors._id: 677229bf8103ad52cb7031b5
paper_authors.name: Junfu Pu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-30T20:42:39.399Z
paper_authors._id: 646c5fcc97819a8be938ee9b
paper_authors.avatarUrl: /avatars/27976d701e88aebf5855418c055ba50e.svg
paper_authors.fullname: Junfu Pu
paper_authors.type: user
paper_authors.user: Jevin754
paper_authors._id: 677229bf8103ad52cb7031b6
paper_authors.name: Huanzhang Dou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-30T20:42:34.026Z
paper_authors._id: 64c8ad3a527d763655600f6f
paper_authors.avatarUrl: /avatars/6a4ab87fe040d666313f59d93712cf73.svg
paper_authors.fullname: Huanzhang Dou
paper_authors.type: user
paper_authors.user: Huanzhang
paper_authors._id: 677229bf8103ad52cb7031b7
paper_authors.name: Guangcong Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-30T20:42:21.249Z
paper_authors._id: 6400b8af330a45b036094a87
paper_authors.avatarUrl: /avatars/1985c994dab49c8d2b7ec07cbea8eb7e.svg
paper_authors.fullname: GuangcongZheng
paper_authors.type: user
paper_authors.user: ZGCTroy
paper_authors._id: 677229bf8103ad52cb7031b8
paper_authors.name: Ying Shan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-30T20:42:27.994Z
paper_authors._id: 63ca3ddc04c979828310bfcb
paper_authors.avatarUrl: /avatars/615e0d8622950b4408b40d550f02a894.svg
paper_authors.fullname: Ying Shan
paper_authors.type: user
paper_authors.user: yshan2u
paper_authors._id: 677229bf8103ad52cb7031b9
paper_authors.name: Xi Li
paper_title: VideoMaker: Zero-shot Customized Video Generation with the Inherent
    Force of Video Diffusion Models
paper_summary: Zero-shot customized video generation has gained significant attention due to
  its substantial application potential. Existing methods rely on additional
  models to extract and inject reference subject features, assuming that the
  Video Diffusion Model (VDM) alone is insufficient for zero-shot customized
  video generation. However, these methods often struggle to maintain consistent
  subject appearance due to suboptimal feature extraction and injection
  techniques. In this paper, we reveal that VDM inherently possesses the force to
  extract and inject subject features. Departing from previous heuristic
  approaches, we introduce a novel framework that leverages VDM's inherent force
  to enable high-quality zero-shot customized video generation. Specifically, for
  feature extraction, we directly input reference images into VDM and use its
  intrinsic feature extraction process, which not only provides fine-grained
  features but also significantly aligns with VDM's pre-trained knowledge. For
  feature injection, we devise an innovative bidirectional interaction between
  subject features and generated content through spatial self-attention within
  VDM, ensuring that VDM has better subject fidelity while maintaining the
  diversity of the generated video.Experiments on both customized human and
  object video generation validate the effectiveness of our framework.
paper_ai_keywords: Video Diffusion Model (VDM) feature extraction feature injection bidirectional interaction spatial self-attention subject fidelity