Still-Moving: Customized Video Generation without Customized Video Data
Customizing text-to-image (T2I) models has seen tremendous progress recently,
particularly in areas such as personalization, stylization, and conditional
generation. However, expanding this progress to video generation is still in
its infancy, primarily due to the lack of customized video data. In this work,
we introduce Still-Moving, a novel generic framework for customizing a
text-to-video (T2V) model, without requiring any customized video data. The
framework applies to the prominent T2V design where the video model is built
over a text-to-image (T2I) model (e.g., via inflation). We assume access to a
customized version of the T2I model, trained only on still image data (e.g.,
using DreamBooth or StyleDrop). Naively plugging in the weights of the
customized T2I model into the T2V model often leads to significant artifacts or
insufficient adherence to the customization data. To overcome this issue, we
train lightweight Spatial Adapters that adjust the features produced
by the injected T2I layers. Importantly, our adapters are trained on
"frozen videos" (i.e., repeated images), constructed from image
samples generated by the customized T2I model. This training is facilitated by
a novel Motion Adapter module, which allows us to train on such
static videos while preserving the motion prior of the video model. At test
time, we remove the Motion Adapter modules and leave in only the trained
Spatial Adapters. This restores the motion prior of the T2V model while
adhering to the spatial prior of the customized T2I model. We demonstrate the
effectiveness of our approach on diverse tasks including personalized,
stylized, and conditional generation. In all evaluated scenarios, our method
seamlessly integrates the spatial prior of the customized T2I model with a
motion prior supplied by the T2V model.
669080323a86663c1e437453
Hila Chefer
claimed_verified
2024-07-12T07:40:16.777Z
6181c72cdcc1df2c9de8a4d8
https://cdn-avatars.huggingface.co/v1/production/uploads/1655248010394-6181c72cdcc1df2c9de8a4d8.jpeg
Hila Chefer
user
Hila
669080323a86663c1e437454
Shiran Zada
669080323a86663c1e437455
Roni Paiss
669080323a86663c1e437456
Ariel Ephrat
669080323a86663c1e437457
Omer Tov
669080323a86663c1e437458
Michael Rubinstein
669080323a86663c1e437459
Lior Wolf
669080323a86663c1e43745a
Tali Dekel
669080323a86663c1e43745b
Tomer Michaeli
669080323a86663c1e43745c
Inbar Mosseri
Still-Moving: Customized Video Generation without Customized Video Data
Customizing text-to-image (T2I) models has seen tremendous progress recently,
particularly in areas such as personalization, stylization, and conditional
generation. However, expanding this progress to video generation is still in
its infancy, primarily due to the lack of customized video data. In this work,
we introduce Still-Moving, a novel generic framework for customizing a
text-to-video (T2V) model, without requiring any customized video data. The
framework applies to the prominent T2V design where the video model is built
over a text-to-image (T2I) model (e.g., via inflation). We assume access to a
customized version of the T2I model, trained only on still image data (e.g.,
using DreamBooth or StyleDrop). Naively plugging in the weights of the
customized T2I model into the T2V model often leads to significant artifacts or
insufficient adherence to the customization data. To overcome this issue, we
train lightweight Spatial Adapters that adjust the features produced
by the injected T2I layers. Importantly, our adapters are trained on
"frozen videos" (i.e., repeated images), constructed from image
samples generated by the customized T2I model. This training is facilitated by
a novel Motion Adapter module, which allows us to train on such
static videos while preserving the motion prior of the video model. At test
time, we remove the Motion Adapter modules and leave in only the trained
Spatial Adapters. This restores the motion prior of the T2V model while
adhering to the spatial prior of the customized T2I model. We demonstrate the
effectiveness of our approach on diverse tasks including personalized,
stylized, and conditional generation. In all evaluated scenarios, our method
seamlessly integrates the spatial prior of the customized T2I model with a
motion prior supplied by the T2V model.
A framework for customizing text-to-video models using existing text-to-image customization without video data, employing Spatial Adapters and a Motion Adapter module.
text-to-image (T2I) models text-to-video (T2V) models Still-Moving Spatial Adapters Motion Adapter frozen videos motion prior spatial prior DreamBooth StyleDrop