TextCraftor: Your Text Encoder Can be Image Quality Controller
Diffusion-based text-to-image generative models, e.g., Stable Diffusion, have
revolutionized the field of content generation, enabling significant
advancements in areas like image editing and video synthesis. Despite their
formidable capabilities, these models are not without their limitations. It is
still challenging to synthesize an image that aligns well with the input text,
and multiple runs with carefully crafted prompts are required to achieve
satisfactory results. To mitigate these limitations, numerous studies have
endeavored to fine-tune the pre-trained diffusion models, i.e., UNet, utilizing
various technologies. Yet, amidst these efforts, a pivotal question of
text-to-image diffusion model training has remained largely unexplored: Is it
possible and feasible to fine-tune the text encoder to improve the performance
of text-to-image diffusion models? Our findings reveal that, instead of
replacing the CLIP text encoder used in Stable Diffusion with other large
language models, we can enhance it through our proposed fine-tuning approach,
TextCraftor, leading to substantial improvements in quantitative benchmarks and
human assessments. Interestingly, our technique also empowers controllable
image generation through the interpolation of different text encoders
fine-tuned with various rewards. We also demonstrate that TextCraftor is
orthogonal to UNet finetuning, and can be combined to further improve
generative quality.
660639f5b85b7b4ea5216354
Yanyu Li
660639f5b85b7b4ea5216355
Xian Liu
claimed_verified
2024-10-30T09:56:51.421Z
6423f10a3fa01ecba6fca861
/avatars/e4b4c7dddc5cf962ade45d8bdfa10b1a.svg
Xian Liu
user
alvinliu0
660639f5b85b7b4ea5216356
Anil Kag
660639f5b85b7b4ea5216357
Ju Hu
660639f5b85b7b4ea5216358
Yerlan Idelbayev
660639f5b85b7b4ea5216359
Dhritiman Sagar
660639f5b85b7b4ea521635a
Yanzhi Wang
660639f5b85b7b4ea521635b
Sergey Tulyakov
660639f5b85b7b4ea521635c
Jian Ren
claimed_verified
2024-04-03T09:22:04.786Z
61f19829233c91cbd2f79e70
/avatars/a0735a94542b4f7cda5aed8bc4be0538.svg
Jian Ren
user
alanspike
TextCraftor: Your Text Encoder Can be Image Quality Controller
Diffusion-based text-to-image generative models, e.g., Stable Diffusion, have
revolutionized the field of content generation, enabling significant
advancements in areas like image editing and video synthesis. Despite their
formidable capabilities, these models are not without their limitations. It is
still challenging to synthesize an image that aligns well with the input text,
and multiple runs with carefully crafted prompts are required to achieve
satisfactory results. To mitigate these limitations, numerous studies have
endeavored to fine-tune the pre-trained diffusion models, i.e., UNet, utilizing
various technologies. Yet, amidst these efforts, a pivotal question of
text-to-image diffusion model training has remained largely unexplored: Is it
possible and feasible to fine-tune the text encoder to improve the performance
of text-to-image diffusion models? Our findings reveal that, instead of
replacing the CLIP text encoder used in Stable Diffusion with other large
language models, we can enhance it through our proposed fine-tuning approach,
TextCraftor, leading to substantial improvements in quantitative benchmarks and
human assessments. Interestingly, our technique also empowers controllable
image generation through the interpolation of different text encoders
fine-tuned with various rewards. We also demonstrate that TextCraftor is
orthogonal to UNet finetuning, and can be combined to further improve
generative quality.
Fine-tuning the CLIP text encoder within diffusion-based text-to-image models using the TextCraftor approach improves performance and enables controllable image generation.
diffusion-based text-to-image generative models Stable Diffusion text-to-image diffusion model training CLIP text encoder fine-tuning TextCraftor controllable image generation interoperability with UNet fine-tuning