Multi-subject Open-set Personalization in Video Generation
Video personalization methods allow us to synthesize videos with specific
concepts such as people, pets, and places. However, existing methods often
focus on limited domains, require time-consuming optimization per subject, or
support only a single subject. We present Video Alchemist - a video model
with built-in multi-subject, open-set personalization capabilities for both
foreground objects and background, eliminating the need for time-consuming
test-time optimization. Our model is built on a new Diffusion Transformer
module that fuses each conditional reference image and its corresponding
subject-level text prompt with cross-attention layers. Developing such a large
model presents two main challenges: dataset and evaluation. First, as paired
datasets of reference images and videos are extremely hard to collect, we
sample selected video frames as reference images and synthesize a clip of the
target video. However, while models can easily denoise training videos given
reference frames, they fail to generalize to new contexts. To mitigate this
issue, we design a new automatic data construction pipeline with extensive
image augmentations. Second, evaluating open-set video personalization is a
challenge in itself. To address this, we introduce a personalization benchmark
that focuses on accurate subject fidelity and supports diverse personalization
scenarios. Finally, our extensive experiments show that our method
significantly outperforms existing personalization methods in both quantitative
and qualitative evaluations.
6784c8f85ffcc11850583e40
Tsai-Shien Chen
admin_assigned
2025-01-13T09:39:01.856Z
6330f570b68c7453d2ee8945
/avatars/5b394a72f277aa6c886b21b477010ab0.svg
Tsai-Shien Chen
user
tschen
6784c8f85ffcc11850583e41
Aliaksandr Siarohin
admin_assigned
2025-01-13T09:39:08.671Z
64276311eb9a0ed86180715b
/avatars/76f933cd549f10e5e2db379de235d304.svg
Aliaksandr Siarohin
user
aliaksandr-siarohin
6784c8f85ffcc11850583e42
Willi Menapace
admin_assigned
2025-01-13T09:39:22.216Z
6315358a362e3e95ea538081
/avatars/3b089a25a87c2e83c6b23ccb5d2dc73e.svg
Willi Menapace
user
willi-menapace
6784c8f85ffcc11850583e43
Yuwei Fang
admin_assigned
2025-01-13T09:39:37.004Z
6406d1dcc3ab325efa9db198
/avatars/781f3cd455596f7cffa0906cc487f79b.svg
Yuwei Fang
user
studyfang
6784c8f85ffcc11850583e44
Kwot Sin Lee
6784c8f85ffcc11850583e45
Ivan Skorokhodov
admin_assigned
2025-01-13T09:39:50.742Z
63610db13c7147fae7de88e3
/avatars/d7e97a16cfee39e1e50d7a5b747876f1.svg
Ivan Skorokhodov
user
universome
6784c8f85ffcc11850583e46
Kfir Aberman
admin_assigned
2025-01-13T09:39:55.987Z
64db29097266618e853dd6ec
https://cdn-avatars.huggingface.co/v1/production/uploads/64db29097266618e853dd6ec/r0MaPQCfAxeKv3ycdKYLK.jpeg
Kfir Aberman
user
kaberman
6784c8f85ffcc11850583e47
Jun-Yan Zhu
admin_assigned
2025-01-13T09:40:02.174Z
63a0acc32fabbbb899952a2b
https://cdn-avatars.huggingface.co/v1/production/uploads/1671474335794-noauth.jpeg
Jun-Yan Zhu
user
junyanz
6784c8f85ffcc11850583e48
Ming-Hsuan Yang
6784c8f85ffcc11850583e49
Sergey Tulyakov
Multi-subject Open-set Personalization in Video Generation
Video personalization methods allow us to synthesize videos with specific
concepts such as people, pets, and places. However, existing methods often
focus on limited domains, require time-consuming optimization per subject, or
support only a single subject. We present Video Alchemist - a video model
with built-in multi-subject, open-set personalization capabilities for both
foreground objects and background, eliminating the need for time-consuming
test-time optimization. Our model is built on a new Diffusion Transformer
module that fuses each conditional reference image and its corresponding
subject-level text prompt with cross-attention layers. Developing such a large
model presents two main challenges: dataset and evaluation. First, as paired
datasets of reference images and videos are extremely hard to collect, we
sample selected video frames as reference images and synthesize a clip of the
target video. However, while models can easily denoise training videos given
reference frames, they fail to generalize to new contexts. To mitigate this
issue, we design a new automatic data construction pipeline with extensive
image augmentations. Second, evaluating open-set video personalization is a
challenge in itself. To address this, we introduce a personalization benchmark
that focuses on accurate subject fidelity and supports diverse personalization
scenarios. Finally, our extensive experiments show that our method
significantly outperforms existing personalization methods in both quantitative
and qualitative evaluations.
Video Alchemist uses a Diffusion Transformer with cross-attention to enable multi-subject personalization without test-time optimization, outperforming existing methods.
Diffusion Transformer cross-attention video personalization subject fidelity automatic data construction pipeline