title: StyleCrafter: Enhancing Stylized Text-to-Video Generation with Style
    Adapter
summary: Text-to-video (T2V) models have shown remarkable capabilities in generating
  diverse videos. However, they struggle to produce user-desired stylized videos
  due to (i) text's inherent clumsiness in expressing specific styles and (ii)
  the generally degraded style fidelity. To address these challenges, we
  introduce StyleCrafter, a generic method that enhances pre-trained T2V models
  with a style control adapter, enabling video generation in any style by
  providing a reference image. Considering the scarcity of stylized video
  datasets, we propose to first train a style control adapter using style-rich
  image datasets, then transfer the learned stylization ability to video
  generation through a tailor-made finetuning paradigm. To promote content-style
  disentanglement, we remove style descriptions from the text prompt and extract
  style information solely from the reference image using a decoupling learning
  strategy. Additionally, we design a scale-adaptive fusion module to balance the
  influences of text-based content features and image-based style features, which
  helps generalization across various text and style combinations. StyleCrafter
  efficiently generates high-quality stylized videos that align with the content
  of the texts and resemble the style of the reference images. Experiments
  demonstrate that our approach is more flexible and efficient than existing
  competitors.
paper_authors._id: 656d37f1e8bf55919ab94676
paper_authors.name: Gongye Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-04T10:02:03.976Z
paper_authors._id: 6553316bf151de82f6a23e1d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6553316bf151de82f6a23e1d/GTBkSj4Fa3OoyM6Muz_Sc.jpeg
paper_authors.fullname: Gongye Liu
paper_authors.type: user
paper_authors.user: liuhuohuo
paper_authors._id: 656d37f1e8bf55919ab94677
paper_authors.name: Menghan Xia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-04T10:02:14.170Z
paper_authors._id: 63401c89f81b9d101361f712
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665146415483-63401c89f81b9d101361f712.png
paper_authors.fullname: Richard
paper_authors.type: user
paper_authors.user: menghanxia
paper_authors._id: 656d37f1e8bf55919ab94678
paper_authors.name: Yong Zhang
paper_authors._id: 656d37f1e8bf55919ab94679
paper_authors.name: Haoxin Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-19T09:23:25.731Z
paper_authors._id: 6305cb2f435ec751b723d80a
paper_authors.avatarUrl: /avatars/5e8519d9814038726bf98f4104f74568.svg
paper_authors.fullname: Haoxin Chen
paper_authors.type: user
paper_authors.user: paulai
paper_authors._id: 656d37f1e8bf55919ab9467a
paper_authors.name: Jinbo Xing
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-04T10:02:57.294Z
paper_authors._id: 64770e86d7cf39f2e937ae9a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64770e86d7cf39f2e937ae9a/pLqGg2z1KzQxCGpMwds-9.jpeg
paper_authors.fullname: Jinbo Xing
paper_authors.type: user
paper_authors.user: Doubiiu
paper_authors._id: 656d37f1e8bf55919ab9467b
paper_authors.name: Xintao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-04T10:03:16.203Z
paper_authors._id: 60e272ca6c78a8c122b12127
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60e272ca6c78a8c122b12127/xldEGBzGrU-bX6IwAw0Ie.jpeg
paper_authors.fullname: Xintao Wang
paper_authors.type: user
paper_authors.user: Xintao
paper_authors._id: 656d37f1e8bf55919ab9467c
paper_authors.name: Yujiu Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-04T10:03:23.236Z
paper_authors._id: 64ca1fe838837b12d5e529b7
paper_authors.avatarUrl: /avatars/44a3ad9e59318784ac531993b5f69f6b.svg
paper_authors.fullname: Yujiu Yang
paper_authors.type: user
paper_authors.user: Thu-redrobot
paper_authors._id: 656d37f1e8bf55919ab9467d
paper_authors.name: Ying Shan
paper_title: StyleCrafter: Enhancing Stylized Text-to-Video Generation with Style
    Adapter
paper_summary: Text-to-video (T2V) models have shown remarkable capabilities in generating
  diverse videos. However, they struggle to produce user-desired stylized videos
  due to (i) text's inherent clumsiness in expressing specific styles and (ii)
  the generally degraded style fidelity. To address these challenges, we
  introduce StyleCrafter, a generic method that enhances pre-trained T2V models
  with a style control adapter, enabling video generation in any style by
  providing a reference image. Considering the scarcity of stylized video
  datasets, we propose to first train a style control adapter using style-rich
  image datasets, then transfer the learned stylization ability to video
  generation through a tailor-made finetuning paradigm. To promote content-style
  disentanglement, we remove style descriptions from the text prompt and extract
  style information solely from the reference image using a decoupling learning
  strategy. Additionally, we design a scale-adaptive fusion module to balance the
  influences of text-based content features and image-based style features, which
  helps generalization across various text and style combinations. StyleCrafter
  efficiently generates high-quality stylized videos that align with the content
  of the texts and resemble the style of the reference images. Experiments
  demonstrate that our approach is more flexible and efficient than existing
  competitors.
paper_ai_summary: StyleCrafter enhances pre-trained T2V models with a style control adapter using style-rich images to produce high-quality, stylized videos that align with text content.
paper_ai_keywords: StyleCrafter style control adapter text-to-video style-rich images style descriptions decoupling learning scale-adaptive fusion module style transfer text-based content features image-based style features