Subject-driven Video Generation via Disentangled Identity and Motion
We propose to train a subject-driven customized video generation model
through decoupling the subject-specific learning from temporal dynamics in
zero-shot without additional tuning. A traditional method for video
customization that is tuning-free often relies on large, annotated video
datasets, which are computationally expensive and require extensive annotation.
In contrast to the previous approach, we introduce the use of an image
customization dataset directly on training video customization models,
factorizing the video customization into two folds: (1) identity injection
through image customization dataset and (2) temporal modeling preservation with
a small set of unannotated videos through the image-to-video training method.
Additionally, we employ random image token dropping with randomized image
initialization during image-to-video fine-tuning to mitigate the copy-and-paste
issue. To further enhance learning, we introduce stochastic switching during
joint optimization of subject-specific and temporal features, mitigating
catastrophic forgetting. Our method achieves strong subject consistency and
scalability, outperforming existing video customization models in zero-shot
settings, demonstrating the effectiveness of our framework.
680ed2679e529f7799a0689f
Daneul Kim
claimed_verified
2025-04-28T07:38:37.502Z
636b20591340f879a2eb98d0
/avatars/4fc5cb13f916bcbc842ccf387bd5f6c0.svg
Daneul Kim
user
carpedkm
680ed2679e529f7799a068a0
Jingxu Zhang
680ed2679e529f7799a068a1
Wonjoon Jin
admin_assigned
2025-04-29T12:35:17.327Z
65de95e37facd099b5050796
/avatars/62c0887cfa7e1946ff32bcc2f44bf747.svg
Wonjoon Jin
user
Wonjoon-Jin
680ed2679e529f7799a068a2
Sunghyun Cho
admin_assigned
2025-04-29T12:35:24.373Z
675d3d94036615f450fe818f
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Oc4-UMIHyVe0naZ0XGQmj.png
sunghyun cho
user
chosh1110
680ed2679e529f7799a068a3
Qi Dai
extracted_pending
2025-04-28T00:57:12.914Z
65115c00a588fdb36558b673
/avatars/1f36263dc4bfaf696a4aa959a6aab1e1.svg
Qi Dai
user
daiqi
680ed2679e529f7799a068a4
Jaesik Park
680ed2679e529f7799a068a5
Chong Luo
extracted_pending
2025-04-28T00:57:12.914Z
676a328148d749b7086782d0
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Tt7u8l8f_1oVBWmBp7tkm.png
Chong Luo
user
cluo-ms
Subject-driven Video Generation via Disentangled Identity and Motion
We propose to train a subject-driven customized video generation model
through decoupling the subject-specific learning from temporal dynamics in
zero-shot without additional tuning. A traditional method for video
customization that is tuning-free often relies on large, annotated video
datasets, which are computationally expensive and require extensive annotation.
In contrast to the previous approach, we introduce the use of an image
customization dataset directly on training video customization models,
factorizing the video customization into two folds: (1) identity injection
through image customization dataset and (2) temporal modeling preservation with
a small set of unannotated videos through the image-to-video training method.
Additionally, we employ random image token dropping with randomized image
initialization during image-to-video fine-tuning to mitigate the copy-and-paste
issue. To further enhance learning, we introduce stochastic switching during
joint optimization of subject-specific and temporal features, mitigating
catastrophic forgetting. Our method achieves strong subject consistency and
scalability, outperforming existing video customization models in zero-shot
settings, demonstrating the effectiveness of our framework.
A method for zero-shot video customization achieves strong performance by decoupling subject-specific learning from temporal dynamics using image customization datasets and randomization techniques.
identity injection image customization dataset temporal modeling image-to-video training random image token dropping stochastic switching catastrophic forgetting subject consistency scalability zero-shot settings