title: Still-Moving: Customized Video Generation without Customized Video Data
summary: Customizing text-to-image (T2I) models has seen tremendous progress recently,
  particularly in areas such as personalization, stylization, and conditional
  generation. However, expanding this progress to video generation is still in
  its infancy, primarily due to the lack of customized video data. In this work,
  we introduce Still-Moving, a novel generic framework for customizing a
  text-to-video (T2V) model, without requiring any customized video data. The
  framework applies to the prominent T2V design where the video model is built
  over a text-to-image (T2I) model (e.g., via inflation). We assume access to a
  customized version of the T2I model, trained only on still image data (e.g.,
  using DreamBooth or StyleDrop). Naively plugging in the weights of the
  customized T2I model into the T2V model often leads to significant artifacts or
  insufficient adherence to the customization data. To overcome this issue, we
  train lightweight Spatial Adapters that adjust the features produced
  by the injected T2I layers. Importantly, our adapters are trained on
  "frozen videos" (i.e., repeated images), constructed from image
  samples generated by the customized T2I model. This training is facilitated by
  a novel Motion Adapter module, which allows us to train on such
  static videos while preserving the motion prior of the video model. At test
  time, we remove the Motion Adapter modules and leave in only the trained
  Spatial Adapters. This restores the motion prior of the T2V model while
  adhering to the spatial prior of the customized T2I model. We demonstrate the
  effectiveness of our approach on diverse tasks including personalized,
  stylized, and conditional generation. In all evaluated scenarios, our method
  seamlessly integrates the spatial prior of the customized T2I model with a
  motion prior supplied by the T2V model.
paper_authors._id: 669080323a86663c1e437453
paper_authors.name: Hila Chefer
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-12T07:40:16.777Z
paper_authors._id: 6181c72cdcc1df2c9de8a4d8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1655248010394-6181c72cdcc1df2c9de8a4d8.jpeg
paper_authors.fullname: Hila Chefer
paper_authors.type: user
paper_authors.user: Hila
paper_authors._id: 669080323a86663c1e437454
paper_authors.name: Shiran Zada
paper_authors._id: 669080323a86663c1e437455
paper_authors.name: Roni Paiss
paper_authors._id: 669080323a86663c1e437456
paper_authors.name: Ariel Ephrat
paper_authors._id: 669080323a86663c1e437457
paper_authors.name: Omer Tov
paper_authors._id: 669080323a86663c1e437458
paper_authors.name: Michael Rubinstein
paper_authors._id: 669080323a86663c1e437459
paper_authors.name: Lior Wolf
paper_authors._id: 669080323a86663c1e43745a
paper_authors.name: Tali Dekel
paper_authors._id: 669080323a86663c1e43745b
paper_authors.name: Tomer Michaeli
paper_authors._id: 669080323a86663c1e43745c
paper_authors.name: Inbar Mosseri
paper_title: Still-Moving: Customized Video Generation without Customized Video Data
paper_summary: Customizing text-to-image (T2I) models has seen tremendous progress recently,
  particularly in areas such as personalization, stylization, and conditional
  generation. However, expanding this progress to video generation is still in
  its infancy, primarily due to the lack of customized video data. In this work,
  we introduce Still-Moving, a novel generic framework for customizing a
  text-to-video (T2V) model, without requiring any customized video data. The
  framework applies to the prominent T2V design where the video model is built
  over a text-to-image (T2I) model (e.g., via inflation). We assume access to a
  customized version of the T2I model, trained only on still image data (e.g.,
  using DreamBooth or StyleDrop). Naively plugging in the weights of the
  customized T2I model into the T2V model often leads to significant artifacts or
  insufficient adherence to the customization data. To overcome this issue, we
  train lightweight Spatial Adapters that adjust the features produced
  by the injected T2I layers. Importantly, our adapters are trained on
  "frozen videos" (i.e., repeated images), constructed from image
  samples generated by the customized T2I model. This training is facilitated by
  a novel Motion Adapter module, which allows us to train on such
  static videos while preserving the motion prior of the video model. At test
  time, we remove the Motion Adapter modules and leave in only the trained
  Spatial Adapters. This restores the motion prior of the T2V model while
  adhering to the spatial prior of the customized T2I model. We demonstrate the
  effectiveness of our approach on diverse tasks including personalized,
  stylized, and conditional generation. In all evaluated scenarios, our method
  seamlessly integrates the spatial prior of the customized T2I model with a
  motion prior supplied by the T2V model.
paper_ai_keywords: text-to-image (T2I) models text-to-video (T2V) models personalization stylization conditional generation Still-Moving framework inflation DreamBooth StyleDrop Spatial Adapters frozen videos Motion Adapter module motion prior spatial prior