SUGAR: Subject-Driven Video Customization in a Zero-Shot Manner
We present SUGAR, a zero-shot method for subject-driven video customization.
Given an input image, SUGAR is capable of generating videos for the subject
contained in the image and aligning the generation with arbitrary visual
attributes such as style and motion specified by user-input text. Unlike
previous methods, which require test-time fine-tuning or fail to generate
text-aligned videos, SUGAR achieves superior results without the need for extra
cost at test-time. To enable zero-shot capability, we introduce a scalable
pipeline to construct synthetic dataset which is specifically designed for
subject-driven customization, leading to 2.5 millions of image-video-text
triplets. Additionally, we propose several methods to enhance our model,
including special attention designs, improved training strategies, and a
refined sampling algorithm. Extensive experiments are conducted. Compared to
previous methods, SUGAR achieves state-of-the-art results in identity
preservation, video dynamics, and video-text alignment for subject-driven video
customization, demonstrating the effectiveness of our proposed method.
676378e1cb7c17a17a5d6f76
Yufan Zhou
claimed_verified
2024-12-19T18:01:46.249Z
635c2c2a7a165601151d3f85
https://cdn-avatars.huggingface.co/v1/production/uploads/1666984965757-noauth.png
Yufan Zhou
user
YfZ
676378e1cb7c17a17a5d6f77
Ruiyi Zhang
676378e1cb7c17a17a5d6f78
Jiuxiang Gu
676378e1cb7c17a17a5d6f79
Nanxuan Zhao
676378e1cb7c17a17a5d6f7a
Jing Shi
676378e1cb7c17a17a5d6f7b
Tong Sun
SUGAR: Subject-Driven Video Customization in a Zero-Shot Manner
We present SUGAR, a zero-shot method for subject-driven video customization.
Given an input image, SUGAR is capable of generating videos for the subject
contained in the image and aligning the generation with arbitrary visual
attributes such as style and motion specified by user-input text. Unlike
previous methods, which require test-time fine-tuning or fail to generate
text-aligned videos, SUGAR achieves superior results without the need for extra
cost at test-time. To enable zero-shot capability, we introduce a scalable
pipeline to construct synthetic dataset which is specifically designed for
subject-driven customization, leading to 2.5 millions of image-video-text
triplets. Additionally, we propose several methods to enhance our model,
including special attention designs, improved training strategies, and a
refined sampling algorithm. Extensive experiments are conducted. Compared to
previous methods, SUGAR achieves state-of-the-art results in identity
preservation, video dynamics, and video-text alignment for subject-driven video
customization, demonstrating the effectiveness of our proposed method.
SUGAR is a zero-shot method for subject-driven video customization that generates videos with specified style and motion attributes without test-time fine-tuning, achieving state-of-the-art results.
zero-shot method subject-driven video customization image-video-text triplets special attention designs improved training strategies refined sampling algorithm identity preservation video dynamics video-text alignment