title: SUGAR: Subject-Driven Video Customization in a Zero-Shot Manner
summary: We present SUGAR, a zero-shot method for subject-driven video customization.
  Given an input image, SUGAR is capable of generating videos for the subject
  contained in the image and aligning the generation with arbitrary visual
  attributes such as style and motion specified by user-input text. Unlike
  previous methods, which require test-time fine-tuning or fail to generate
  text-aligned videos, SUGAR achieves superior results without the need for extra
  cost at test-time. To enable zero-shot capability, we introduce a scalable
  pipeline to construct synthetic dataset which is specifically designed for
  subject-driven customization, leading to 2.5 millions of image-video-text
  triplets. Additionally, we propose several methods to enhance our model,
  including special attention designs, improved training strategies, and a
  refined sampling algorithm. Extensive experiments are conducted. Compared to
  previous methods, SUGAR achieves state-of-the-art results in identity
  preservation, video dynamics, and video-text alignment for subject-driven video
  customization, demonstrating the effectiveness of our proposed method.
paper_authors._id: 676378e1cb7c17a17a5d6f76
paper_authors.name: Yufan Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-19T18:01:46.249Z
paper_authors._id: 635c2c2a7a165601151d3f85
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666984965757-noauth.png
paper_authors.fullname: Yufan Zhou
paper_authors.type: user
paper_authors.user: YfZ
paper_authors._id: 676378e1cb7c17a17a5d6f77
paper_authors.name: Ruiyi Zhang
paper_authors._id: 676378e1cb7c17a17a5d6f78
paper_authors.name: Jiuxiang Gu
paper_authors._id: 676378e1cb7c17a17a5d6f79
paper_authors.name: Nanxuan Zhao
paper_authors._id: 676378e1cb7c17a17a5d6f7a
paper_authors.name: Jing Shi
paper_authors._id: 676378e1cb7c17a17a5d6f7b
paper_authors.name: Tong Sun
paper_title: SUGAR: Subject-Driven Video Customization in a Zero-Shot Manner
paper_summary: We present SUGAR, a zero-shot method for subject-driven video customization.
  Given an input image, SUGAR is capable of generating videos for the subject
  contained in the image and aligning the generation with arbitrary visual
  attributes such as style and motion specified by user-input text. Unlike
  previous methods, which require test-time fine-tuning or fail to generate
  text-aligned videos, SUGAR achieves superior results without the need for extra
  cost at test-time. To enable zero-shot capability, we introduce a scalable
  pipeline to construct synthetic dataset which is specifically designed for
  subject-driven customization, leading to 2.5 millions of image-video-text
  triplets. Additionally, we propose several methods to enhance our model,
  including special attention designs, improved training strategies, and a
  refined sampling algorithm. Extensive experiments are conducted. Compared to
  previous methods, SUGAR achieves state-of-the-art results in identity
  preservation, video dynamics, and video-text alignment for subject-driven video
  customization, demonstrating the effectiveness of our proposed method.
paper_ai_summary: SUGAR is a zero-shot method for subject-driven video customization that generates videos with specified style and motion attributes without test-time fine-tuning, achieving state-of-the-art results.
paper_ai_keywords: zero-shot method subject-driven video customization image-video-text triplets special attention designs improved training strategies refined sampling algorithm identity preservation video dynamics video-text alignment