title: CustomCrafter: Customized Video Generation with Preserving Motion and
    Concept Composition Abilities
summary: Customized video generation aims to generate high-quality videos guided by
  text prompts and subject's reference images. However, since it is only trained
  on static images, the fine-tuning process of subject learning disrupts
  abilities of video diffusion models (VDMs) to combine concepts and generate
  motions. To restore these abilities, some methods use additional video similar
  to the prompt to fine-tune or guide the model. This requires frequent changes
  of guiding videos and even re-tuning of the model when generating different
  motions, which is very inconvenient for users. In this paper, we propose
  CustomCrafter, a novel framework that preserves the model's motion generation
  and conceptual combination abilities without additional video and fine-tuning
  to recovery. For preserving conceptual combination ability, we design a
  plug-and-play module to update few parameters in VDMs, enhancing the model's
  ability to capture the appearance details and the ability of concept
  combinations for new subjects. For motion generation, we observed that VDMs
  tend to restore the motion of video in the early stage of denoising, while
  focusing on the recovery of subject details in the later stage. Therefore, we
  propose Dynamic Weighted Video Sampling Strategy. Using the pluggability of our
  subject learning modules, we reduce the impact of this module on motion
  generation in the early stage of denoising, preserving the ability to generate
  motion of VDMs. In the later stage of denoising, we restore this module to
  repair the appearance details of the specified subject, thereby ensuring the
  fidelity of the subject's appearance. Experimental results show that our method
  has a significant improvement compared to previous methods.
paper_authors._id: 66cbe22f8347e21e8a75a852
paper_authors.name: Tao Wu
paper_authors._id: 66cbe22f8347e21e8a75a853
paper_authors.name: Yong Zhang
paper_authors._id: 66cbe22f8347e21e8a75a854
paper_authors.name: Xintao Wang
paper_authors._id: 66cbe22f8347e21e8a75a855
paper_authors.name: Xianpan Zhou
paper_authors._id: 66cbe22f8347e21e8a75a856
paper_authors.name: Guangcong Zheng
paper_authors._id: 66cbe22f8347e21e8a75a857
paper_authors.name: Zhongang Qi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-19T08:40:08.908Z
paper_authors._id: 660cc64f1b14f691990c0ea0
paper_authors.avatarUrl: /avatars/f172d3120b22d745a41cc3f2eb499ce6.svg
paper_authors.fullname: Zhongang Qi
paper_authors.type: user
paper_authors.user: phoenixqza
paper_authors._id: 66cbe22f8347e21e8a75a858
paper_authors.name: Ying Shan
paper_authors._id: 66cbe22f8347e21e8a75a859
paper_authors.name: Xi Li
paper_title: CustomCrafter: Customized Video Generation with Preserving Motion and
    Concept Composition Abilities
paper_summary: Customized video generation aims to generate high-quality videos guided by
  text prompts and subject's reference images. However, since it is only trained
  on static images, the fine-tuning process of subject learning disrupts
  abilities of video diffusion models (VDMs) to combine concepts and generate
  motions. To restore these abilities, some methods use additional video similar
  to the prompt to fine-tune or guide the model. This requires frequent changes
  of guiding videos and even re-tuning of the model when generating different
  motions, which is very inconvenient for users. In this paper, we propose
  CustomCrafter, a novel framework that preserves the model's motion generation
  and conceptual combination abilities without additional video and fine-tuning
  to recovery. For preserving conceptual combination ability, we design a
  plug-and-play module to update few parameters in VDMs, enhancing the model's
  ability to capture the appearance details and the ability of concept
  combinations for new subjects. For motion generation, we observed that VDMs
  tend to restore the motion of video in the early stage of denoising, while
  focusing on the recovery of subject details in the later stage. Therefore, we
  propose Dynamic Weighted Video Sampling Strategy. Using the pluggability of our
  subject learning modules, we reduce the impact of this module on motion
  generation in the early stage of denoising, preserving the ability to generate
  motion of VDMs. In the later stage of denoising, we restore this module to
  repair the appearance details of the specified subject, thereby ensuring the
  fidelity of the subject's appearance. Experimental results show that our method
  has a significant improvement compared to previous methods.
paper_ai_keywords: video diffusion models (VDMs) fine-tuning subject learning plug-and-play module parameter update Dynamic Weighted Video Sampling Strategy denoising motion generation appearance details concept combinations experimental results