title: CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept
    Matching
summary: Diffusion models have demonstrated great success in the field of
  text-to-image generation. However, alleviating the misalignment between the
  text prompts and images is still challenging. The root reason behind the
  misalignment has not been extensively investigated. We observe that the
  misalignment is caused by inadequate token attention activation. We further
  attribute this phenomenon to the diffusion model's insufficient condition
  utilization, which is caused by its training paradigm. To address the issue, we
  propose CoMat, an end-to-end diffusion model fine-tuning strategy with an
  image-to-text concept matching mechanism. We leverage an image captioning model
  to measure image-to-text alignment and guide the diffusion model to revisit
  ignored tokens. A novel attribute concentration module is also proposed to
  address the attribute binding problem. Without any image or human preference
  data, we use only 20K text prompts to fine-tune SDXL to obtain CoMat-SDXL.
  Extensive experiments show that CoMat-SDXL significantly outperforms the
  baseline model SDXL in two text-to-image alignment benchmarks and achieves
  start-of-the-art performance.
paper_authors._id: 660f613059d8544c00e52eab
paper_authors.name: Dongzhi Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-05T10:13:50.800Z
paper_authors._id: 6349214f8146350b3a4c5cdf
paper_authors.avatarUrl: /avatars/cfd24caac9a87efb528d0f4c375932bc.svg
paper_authors.fullname: Dongzhi Jiang
paper_authors.type: user
paper_authors.user: CaraJ
paper_authors._id: 660f613059d8544c00e52eac
paper_authors.name: Guanglu Song
paper_authors._id: 660f613059d8544c00e52ead
paper_authors.name: Xiaoshi Wu
paper_authors._id: 660f613059d8544c00e52eae
paper_authors.name: Renrui Zhang
paper_authors._id: 660f613059d8544c00e52eaf
paper_authors.name: Dazhong Shen
paper_authors._id: 660f613059d8544c00e52eb0
paper_authors.name: Zhuofan Zong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T13:44:17.566Z
paper_authors._id: 64db05f23725f8d9a9f566c4
paper_authors.avatarUrl: /avatars/345b6d2dfadb91ac1f02be023dd27c8f.svg
paper_authors.fullname: Zhuofan Zong
paper_authors.type: user
paper_authors.user: zongzhuofan
paper_authors._id: 660f613059d8544c00e52eb1
paper_authors.name: Yu Liu
paper_authors._id: 660f613059d8544c00e52eb2
paper_authors.name: Hongsheng Li
paper_title: CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept
    Matching
paper_summary: Diffusion models have demonstrated great success in the field of
  text-to-image generation. However, alleviating the misalignment between the
  text prompts and images is still challenging. The root reason behind the
  misalignment has not been extensively investigated. We observe that the
  misalignment is caused by inadequate token attention activation. We further
  attribute this phenomenon to the diffusion model's insufficient condition
  utilization, which is caused by its training paradigm. To address the issue, we
  propose CoMat, an end-to-end diffusion model fine-tuning strategy with an
  image-to-text concept matching mechanism. We leverage an image captioning model
  to measure image-to-text alignment and guide the diffusion model to revisit
  ignored tokens. A novel attribute concentration module is also proposed to
  address the attribute binding problem. Without any image or human preference
  data, we use only 20K text prompts to fine-tune SDXL to obtain CoMat-SDXL.
  Extensive experiments show that CoMat-SDXL significantly outperforms the
  baseline model SDXL in two text-to-image alignment benchmarks and achieves
  start-of-the-art performance.
paper_ai_summary: CoMat, an end-to-end diffusion model fine-tuning strategy with an image-to-text concept matching mechanism, improves text-to-image alignment in SDXL without additional data.
paper_ai_keywords: diffusion models text-to-image generation token attention activation condition utilization training paradigm image captioning model attribute concentration module