title: Magic-Me: Identity-Specific Video Customized Diffusion
summary: Creating content for a specific identity (ID) has shown significant interest
  in the field of generative models. In the field of text-to-image generation
  (T2I), subject-driven content generation has achieved great progress with the
  ID in the images controllable. However, extending it to video generation is not
  well explored. In this work, we propose a simple yet effective subject identity
  controllable video generation framework, termed Video Custom Diffusion (VCD).
  With a specified subject ID defined by a few images, VCD reinforces the
  identity information extraction and injects frame-wise correlation at the
  initialization stage for stable video outputs with identity preserved to a
  large extent. To achieve this, we propose three novel components that are
  essential for high-quality ID preservation: 1) an ID module trained with the
  cropped identity by prompt-to-segmentation to disentangle the ID information
  and the background noise for more accurate ID token learning; 2) a
  text-to-video (T2V) VCD module with 3D Gaussian Noise Prior for better
  inter-frame consistency and 3) video-to-video (V2V) Face VCD and Tiled VCD
  modules to deblur the face and upscale the video for higher resolution.
    Despite its simplicity, we conducted extensive experiments to verify that VCD
  is able to generate stable and high-quality videos with better ID over the
  selected strong baselines. Besides, due to the transferability of the ID
  module, VCD is also working well with finetuned text-to-image models available
  publically, further improving its usability. The codes are available at
  https://github.com/Zhen-Dong/Magic-Me.
paper_authors._id: 65cd79e23f97d5bd5d51c04a
paper_authors.name: Ze Ma
paper_authors._id: 65cd79e23f97d5bd5d51c04b
paper_authors.name: Daquan Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-15T08:50:24.450Z
paper_authors._id: 63fe0b160c1bbe8e29d2dd32
paper_authors.avatarUrl: /avatars/bc574036287170a77057893efaa48e2d.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: DaQuan21
paper_authors._id: 65cd79e23f97d5bd5d51c04c
paper_authors.name: Chun-Hsiao Yeh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-15T08:50:31.526Z
paper_authors._id: 6499eca0685215f7247bd5ce
paper_authors.avatarUrl: /avatars/b6fea0c33c3c930c7314b99b414072a9.svg
paper_authors.fullname: Chun-Hsiao Yeh
paper_authors.type: user
paper_authors.user: danielchyeh
paper_authors._id: 65cd79e23f97d5bd5d51c04d
paper_authors.name: Xue-She Wang
paper_authors._id: 65cd79e23f97d5bd5d51c04e
paper_authors.name: Xiuyu Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-15T08:50:49.215Z
paper_authors._id: 644570ba2d91b15b4c7f6311
paper_authors.avatarUrl: /avatars/d5e66012066d0c330b8f23718b1499d8.svg
paper_authors.fullname: Xiuyu Li
paper_authors.type: user
paper_authors.user: xiuyul
paper_authors._id: 65cd79e23f97d5bd5d51c04f
paper_authors.name: Huanrui Yang
paper_authors._id: 65cd79e23f97d5bd5d51c050
paper_authors.name: Zhen Dong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-15T08:26:22.160Z
paper_authors._id: 643ba2f725681c3afaa8f05e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643ba2f725681c3afaa8f05e/2RnOdmbBM8WHYhiTGG-Cd.jpeg
paper_authors.fullname: Zhen Dong
paper_authors.type: user
paper_authors.user: zhendongucb
paper_authors._id: 65cd79e23f97d5bd5d51c051
paper_authors.name: Kurt Keutzer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-15T08:51:04.156Z
paper_authors._id: 6251bf4b183aa4266924ad91
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678041834400-6251bf4b183aa4266924ad91.jpeg
paper_authors.fullname: Kurt Keutzer
paper_authors.type: user
paper_authors.user: kurtkeutzer
paper_authors._id: 65cd79e23f97d5bd5d51c052
paper_authors.name: Jiashi Feng
paper_title: Magic-Me: Identity-Specific Video Customized Diffusion
paper_summary: Creating content for a specific identity (ID) has shown significant interest
  in the field of generative models. In the field of text-to-image generation
  (T2I), subject-driven content generation has achieved great progress with the
  ID in the images controllable. However, extending it to video generation is not
  well explored. In this work, we propose a simple yet effective subject identity
  controllable video generation framework, termed Video Custom Diffusion (VCD).
  With a specified subject ID defined by a few images, VCD reinforces the
  identity information extraction and injects frame-wise correlation at the
  initialization stage for stable video outputs with identity preserved to a
  large extent. To achieve this, we propose three novel components that are
  essential for high-quality ID preservation: 1) an ID module trained with the
  cropped identity by prompt-to-segmentation to disentangle the ID information
  and the background noise for more accurate ID token learning; 2) a
  text-to-video (T2V) VCD module with 3D Gaussian Noise Prior for better
  inter-frame consistency and 3) video-to-video (V2V) Face VCD and Tiled VCD
  modules to deblur the face and upscale the video for higher resolution.
    Despite its simplicity, we conducted extensive experiments to verify that VCD
  is able to generate stable and high-quality videos with better ID over the
  selected strong baselines. Besides, due to the transferability of the ID
  module, VCD is also working well with finetuned text-to-image models available
  publically, further improving its usability. The codes are available at
  https://github.com/Zhen-Dong/Magic-Me.
paper_ai_summary: A framework named Video Custom Diffusion (VCD) is proposed for subject identity controllable video generation, achieving stable and high-quality outputs with better identity preservation compared to existing methods.
paper_ai_keywords: Video Custom Diffusion VCD ID module prompt-to-segmentation ID token learning text-to-video T2V 3D Gaussian Noise Prior video-to-video V2V.Face VCD Tiled VCD deblur upscale