CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept
  Matching
Diffusion models have demonstrated great success in the field of
text-to-image generation. However, alleviating the misalignment between the
text prompts and images is still challenging. The root reason behind the
misalignment has not been extensively investigated. We observe that the
misalignment is caused by inadequate token attention activation. We further
attribute this phenomenon to the diffusion model's insufficient condition
utilization, which is caused by its training paradigm. To address the issue, we
propose CoMat, an end-to-end diffusion model fine-tuning strategy with an
image-to-text concept matching mechanism. We leverage an image captioning model
to measure image-to-text alignment and guide the diffusion model to revisit
ignored tokens. A novel attribute concentration module is also proposed to
address the attribute binding problem. Without any image or human preference
data, we use only 20K text prompts to fine-tune SDXL to obtain CoMat-SDXL.
Extensive experiments show that CoMat-SDXL significantly outperforms the
baseline model SDXL in two text-to-image alignment benchmarks and achieves
start-of-the-art performance.
660f613059d8544c00e52eab
Dongzhi Jiang
claimed_verified
2024-04-05T10:13:50.800Z
6349214f8146350b3a4c5cdf
/avatars/cfd24caac9a87efb528d0f4c375932bc.svg
Dongzhi Jiang
user
CaraJ
660f613059d8544c00e52eac
Guanglu Song
660f613059d8544c00e52ead
Xiaoshi Wu
660f613059d8544c00e52eae
Renrui Zhang
660f613059d8544c00e52eaf
Dazhong Shen
660f613059d8544c00e52eb0
Zhuofan Zong
claimed_verified
2024-06-18T13:44:17.566Z
64db05f23725f8d9a9f566c4
/avatars/345b6d2dfadb91ac1f02be023dd27c8f.svg
Zhuofan Zong
user
zongzhuofan
660f613059d8544c00e52eb1
Yu Liu
660f613059d8544c00e52eb2
Hongsheng Li
CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept
  Matching
Diffusion models have demonstrated great success in the field of
text-to-image generation. However, alleviating the misalignment between the
text prompts and images is still challenging. The root reason behind the
misalignment has not been extensively investigated. We observe that the
misalignment is caused by inadequate token attention activation. We further
attribute this phenomenon to the diffusion model's insufficient condition
utilization, which is caused by its training paradigm. To address the issue, we
propose CoMat, an end-to-end diffusion model fine-tuning strategy with an
image-to-text concept matching mechanism. We leverage an image captioning model
to measure image-to-text alignment and guide the diffusion model to revisit
ignored tokens. A novel attribute concentration module is also proposed to
address the attribute binding problem. Without any image or human preference
data, we use only 20K text prompts to fine-tune SDXL to obtain CoMat-SDXL.
Extensive experiments show that CoMat-SDXL significantly outperforms the
baseline model SDXL in two text-to-image alignment benchmarks and achieves
start-of-the-art performance.
CoMat, an end-to-end diffusion model fine-tuning strategy with an image-to-text concept matching mechanism, improves text-to-image alignment in SDXL without additional data.
diffusion models text-to-image generation token attention activation condition utilization training paradigm image captioning model attribute concentration module