title: TextCraftor: Your Text Encoder Can be Image Quality Controller
summary: Diffusion-based text-to-image generative models, e.g., Stable Diffusion, have
  revolutionized the field of content generation, enabling significant
  advancements in areas like image editing and video synthesis. Despite their
  formidable capabilities, these models are not without their limitations. It is
  still challenging to synthesize an image that aligns well with the input text,
  and multiple runs with carefully crafted prompts are required to achieve
  satisfactory results. To mitigate these limitations, numerous studies have
  endeavored to fine-tune the pre-trained diffusion models, i.e., UNet, utilizing
  various technologies. Yet, amidst these efforts, a pivotal question of
  text-to-image diffusion model training has remained largely unexplored: Is it
  possible and feasible to fine-tune the text encoder to improve the performance
  of text-to-image diffusion models? Our findings reveal that, instead of
  replacing the CLIP text encoder used in Stable Diffusion with other large
  language models, we can enhance it through our proposed fine-tuning approach,
  TextCraftor, leading to substantial improvements in quantitative benchmarks and
  human assessments. Interestingly, our technique also empowers controllable
  image generation through the interpolation of different text encoders
  fine-tuned with various rewards. We also demonstrate that TextCraftor is
  orthogonal to UNet finetuning, and can be combined to further improve
  generative quality.
paper_authors._id: 660639f5b85b7b4ea5216354
paper_authors.name: Yanyu Li
paper_authors._id: 660639f5b85b7b4ea5216355
paper_authors.name: Xian Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:56:51.421Z
paper_authors._id: 6423f10a3fa01ecba6fca861
paper_authors.avatarUrl: /avatars/e4b4c7dddc5cf962ade45d8bdfa10b1a.svg
paper_authors.fullname: Xian Liu
paper_authors.type: user
paper_authors.user: alvinliu0
paper_authors._id: 660639f5b85b7b4ea5216356
paper_authors.name: Anil Kag
paper_authors._id: 660639f5b85b7b4ea5216357
paper_authors.name: Ju Hu
paper_authors._id: 660639f5b85b7b4ea5216358
paper_authors.name: Yerlan Idelbayev
paper_authors._id: 660639f5b85b7b4ea5216359
paper_authors.name: Dhritiman Sagar
paper_authors._id: 660639f5b85b7b4ea521635a
paper_authors.name: Yanzhi Wang
paper_authors._id: 660639f5b85b7b4ea521635b
paper_authors.name: Sergey Tulyakov
paper_authors._id: 660639f5b85b7b4ea521635c
paper_authors.name: Jian Ren
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-03T09:22:04.786Z
paper_authors._id: 61f19829233c91cbd2f79e70
paper_authors.avatarUrl: /avatars/a0735a94542b4f7cda5aed8bc4be0538.svg
paper_authors.fullname: Jian Ren
paper_authors.type: user
paper_authors.user: alanspike
paper_title: TextCraftor: Your Text Encoder Can be Image Quality Controller
paper_summary: Diffusion-based text-to-image generative models, e.g., Stable Diffusion, have
  revolutionized the field of content generation, enabling significant
  advancements in areas like image editing and video synthesis. Despite their
  formidable capabilities, these models are not without their limitations. It is
  still challenging to synthesize an image that aligns well with the input text,
  and multiple runs with carefully crafted prompts are required to achieve
  satisfactory results. To mitigate these limitations, numerous studies have
  endeavored to fine-tune the pre-trained diffusion models, i.e., UNet, utilizing
  various technologies. Yet, amidst these efforts, a pivotal question of
  text-to-image diffusion model training has remained largely unexplored: Is it
  possible and feasible to fine-tune the text encoder to improve the performance
  of text-to-image diffusion models? Our findings reveal that, instead of
  replacing the CLIP text encoder used in Stable Diffusion with other large
  language models, we can enhance it through our proposed fine-tuning approach,
  TextCraftor, leading to substantial improvements in quantitative benchmarks and
  human assessments. Interestingly, our technique also empowers controllable
  image generation through the interpolation of different text encoders
  fine-tuned with various rewards. We also demonstrate that TextCraftor is
  orthogonal to UNet finetuning, and can be combined to further improve
  generative quality.
paper_ai_summary: Fine-tuning the CLIP text encoder within diffusion-based text-to-image models using the TextCraftor approach improves performance and enables controllable image generation.
paper_ai_keywords: diffusion-based text-to-image generative models Stable Diffusion text-to-image diffusion model training CLIP text encoder fine-tuning TextCraftor controllable image generation interoperability with UNet fine-tuning