VideoMaker: Zero-shot Customized Video Generation with the Inherent
  Force of Video Diffusion Models
Zero-shot customized video generation has gained significant attention due to
its substantial application potential. Existing methods rely on additional
models to extract and inject reference subject features, assuming that the
Video Diffusion Model (VDM) alone is insufficient for zero-shot customized
video generation. However, these methods often struggle to maintain consistent
subject appearance due to suboptimal feature extraction and injection
techniques. In this paper, we reveal that VDM inherently possesses the force to
extract and inject subject features. Departing from previous heuristic
approaches, we introduce a novel framework that leverages VDM's inherent force
to enable high-quality zero-shot customized video generation. Specifically, for
feature extraction, we directly input reference images into VDM and use its
intrinsic feature extraction process, which not only provides fine-grained
features but also significantly aligns with VDM's pre-trained knowledge. For
feature injection, we devise an innovative bidirectional interaction between
subject features and generated content through spatial self-attention within
VDM, ensuring that VDM has better subject fidelity while maintaining the
diversity of the generated video.Experiments on both customized human and
object video generation validate the effectiveness of our framework.
677229bf8103ad52cb7031b1
Tao Wu
677229bf8103ad52cb7031b2
Yong Zhang
677229bf8103ad52cb7031b3
Xiaodong Cun
admin_assigned
2024-12-30T20:42:51.507Z
63184c517ca1b876d99b7e0e
https://cdn-avatars.huggingface.co/v1/production/uploads/63184c517ca1b876d99b7e0e/b-qDExoeJuDXK0cJBZKnz.jpeg
Xiaodong Cun
user
vinthony
677229bf8103ad52cb7031b4
Zhongang Qi
admin_assigned
2024-12-30T20:42:45.157Z
660cc64f1b14f691990c0ea0
/avatars/f172d3120b22d745a41cc3f2eb499ce6.svg
Zhongang Qi
user
phoenixqza
677229bf8103ad52cb7031b5
Junfu Pu
admin_assigned
2024-12-30T20:42:39.399Z
646c5fcc97819a8be938ee9b
/avatars/27976d701e88aebf5855418c055ba50e.svg
Junfu Pu
user
Jevin754
677229bf8103ad52cb7031b6
Huanzhang Dou
admin_assigned
2024-12-30T20:42:34.026Z
64c8ad3a527d763655600f6f
/avatars/6a4ab87fe040d666313f59d93712cf73.svg
Huanzhang Dou
user
Huanzhang
677229bf8103ad52cb7031b7
Guangcong Zheng
admin_assigned
2024-12-30T20:42:21.249Z
6400b8af330a45b036094a87
/avatars/1985c994dab49c8d2b7ec07cbea8eb7e.svg
GuangcongZheng
user
ZGCTroy
677229bf8103ad52cb7031b8
Ying Shan
admin_assigned
2024-12-30T20:42:27.994Z
63ca3ddc04c979828310bfcb
/avatars/615e0d8622950b4408b40d550f02a894.svg
Ying Shan
user
yshan2u
677229bf8103ad52cb7031b9
Xi Li
VideoMaker: Zero-shot Customized Video Generation with the Inherent
  Force of Video Diffusion Models
Zero-shot customized video generation has gained significant attention due to
its substantial application potential. Existing methods rely on additional
models to extract and inject reference subject features, assuming that the
Video Diffusion Model (VDM) alone is insufficient for zero-shot customized
video generation. However, these methods often struggle to maintain consistent
subject appearance due to suboptimal feature extraction and injection
techniques. In this paper, we reveal that VDM inherently possesses the force to
extract and inject subject features. Departing from previous heuristic
approaches, we introduce a novel framework that leverages VDM's inherent force
to enable high-quality zero-shot customized video generation. Specifically, for
feature extraction, we directly input reference images into VDM and use its
intrinsic feature extraction process, which not only provides fine-grained
features but also significantly aligns with VDM's pre-trained knowledge. For
feature injection, we devise an innovative bidirectional interaction between
subject features and generated content through spatial self-attention within
VDM, ensuring that VDM has better subject fidelity while maintaining the
diversity of the generated video.Experiments on both customized human and
object video generation validate the effectiveness of our framework.
A novel framework leverages Video Diffusion Models for zero-shot customized video generation by directly using VDM's intrinsic feature extraction and spatial self-attention for feature injection.
Video Diffusion Model (VDM) feature extraction feature injection bidirectional interaction spatial self-attention