StyleCrafter: Enhancing Stylized Text-to-Video Generation with Style
  Adapter
Text-to-video (T2V) models have shown remarkable capabilities in generating
diverse videos. However, they struggle to produce user-desired stylized videos
due to (i) text's inherent clumsiness in expressing specific styles and (ii)
the generally degraded style fidelity. To address these challenges, we
introduce StyleCrafter, a generic method that enhances pre-trained T2V models
with a style control adapter, enabling video generation in any style by
providing a reference image. Considering the scarcity of stylized video
datasets, we propose to first train a style control adapter using style-rich
image datasets, then transfer the learned stylization ability to video
generation through a tailor-made finetuning paradigm. To promote content-style
disentanglement, we remove style descriptions from the text prompt and extract
style information solely from the reference image using a decoupling learning
strategy. Additionally, we design a scale-adaptive fusion module to balance the
influences of text-based content features and image-based style features, which
helps generalization across various text and style combinations. StyleCrafter
efficiently generates high-quality stylized videos that align with the content
of the texts and resemble the style of the reference images. Experiments
demonstrate that our approach is more flexible and efficient than existing
competitors.
656d37f1e8bf55919ab94676
Gongye Liu
admin_assigned
2023-12-04T10:02:03.976Z
6553316bf151de82f6a23e1d
https://cdn-avatars.huggingface.co/v1/production/uploads/6553316bf151de82f6a23e1d/GTBkSj4Fa3OoyM6Muz_Sc.jpeg
Gongye Liu
user
liuhuohuo
656d37f1e8bf55919ab94677
Menghan Xia
admin_assigned
2023-12-04T10:02:14.170Z
63401c89f81b9d101361f712
https://cdn-avatars.huggingface.co/v1/production/uploads/1665146415483-63401c89f81b9d101361f712.png
Richard
user
menghanxia
656d37f1e8bf55919ab94678
Yong Zhang
656d37f1e8bf55919ab94679
Haoxin Chen
claimed_verified
2024-01-19T09:23:25.731Z
6305cb2f435ec751b723d80a
/avatars/5e8519d9814038726bf98f4104f74568.svg
Haoxin Chen
user
paulai
656d37f1e8bf55919ab9467a
Jinbo Xing
admin_assigned
2023-12-04T10:02:57.294Z
64770e86d7cf39f2e937ae9a
https://cdn-avatars.huggingface.co/v1/production/uploads/64770e86d7cf39f2e937ae9a/pLqGg2z1KzQxCGpMwds-9.jpeg
Jinbo Xing
user
Doubiiu
656d37f1e8bf55919ab9467b
Xintao Wang
admin_assigned
2023-12-04T10:03:16.203Z
60e272ca6c78a8c122b12127
https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
Xintao Wang
user
Xintao
656d37f1e8bf55919ab9467c
Yujiu Yang
admin_assigned
2023-12-04T10:03:23.236Z
64ca1fe838837b12d5e529b7
/avatars/44a3ad9e59318784ac531993b5f69f6b.svg
Yujiu Yang
user
Thu-redrobot
656d37f1e8bf55919ab9467d
Ying Shan
StyleCrafter: Enhancing Stylized Text-to-Video Generation with Style
  Adapter
Text-to-video (T2V) models have shown remarkable capabilities in generating
diverse videos. However, they struggle to produce user-desired stylized videos
due to (i) text's inherent clumsiness in expressing specific styles and (ii)
the generally degraded style fidelity. To address these challenges, we
introduce StyleCrafter, a generic method that enhances pre-trained T2V models
with a style control adapter, enabling video generation in any style by
providing a reference image. Considering the scarcity of stylized video
datasets, we propose to first train a style control adapter using style-rich
image datasets, then transfer the learned stylization ability to video
generation through a tailor-made finetuning paradigm. To promote content-style
disentanglement, we remove style descriptions from the text prompt and extract
style information solely from the reference image using a decoupling learning
strategy. Additionally, we design a scale-adaptive fusion module to balance the
influences of text-based content features and image-based style features, which
helps generalization across various text and style combinations. StyleCrafter
efficiently generates high-quality stylized videos that align with the content
of the texts and resemble the style of the reference images. Experiments
demonstrate that our approach is more flexible and efficient than existing
competitors.
StyleCrafter enhances pre-trained T2V models with a style control adapter using style-rich images to produce high-quality, stylized videos that align with text content.
StyleCrafter style control adapter text-to-video style-rich images style descriptions decoupling learning scale-adaptive fusion module style transfer text-based content features image-based style features