title: Improving Long-Text Alignment for Text-to-Image Diffusion Models
summary: The rapid advancement of text-to-image (T2I) diffusion models has enabled
  them to generate unprecedented results from given texts. However, as text
  inputs become longer, existing encoding methods like CLIP face limitations, and
  aligning the generated images with long texts becomes challenging. To tackle
  these issues, we propose LongAlign, which includes a segment-level encoding
  method for processing long texts and a decomposed preference optimization
  method for effective alignment training. For segment-level encoding, long texts
  are divided into multiple segments and processed separately. This method
  overcomes the maximum input length limits of pretrained encoding models. For
  preference optimization, we provide decomposed CLIP-based preference models to
  fine-tune diffusion models. Specifically, to utilize CLIP-based preference
  models for T2I alignment, we delve into their scoring mechanisms and find that
  the preference scores can be decomposed into two components: a text-relevant
  part that measures T2I alignment and a text-irrelevant part that assesses other
  visual aspects of human preference. Additionally, we find that the
  text-irrelevant part contributes to a common overfitting problem during
  fine-tuning. To address this, we propose a reweighting strategy that assigns
  different weights to these two components, thereby reducing overfitting and
  enhancing alignment. After fine-tuning 512 times 512 Stable Diffusion (SD)
  v1.5 for about 20 hours using our method, the fine-tuned SD outperforms
  stronger foundation models in T2I alignment, such as PixArt-alpha and
  Kandinsky v2.2. The code is available at
  https://github.com/luping-liu/LongAlign.
paper_authors._id: 67106d4b63f0a6c38b2633de
paper_authors.name: Luping Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-17T11:24:54.296Z
paper_authors._id: 63e8a1fc46574e63a2bf3e88
paper_authors.avatarUrl: /avatars/b91f587920c9102cabeca77d72d33995.svg
paper_authors.fullname: Luping Liu
paper_authors.type: user
paper_authors.user: luping-liu
paper_authors._id: 67106d4b63f0a6c38b2633df
paper_authors.name: Chao Du
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:42:59.416Z
paper_authors._id: 632407c892e07e3ca20aca28
paper_authors.avatarUrl: /avatars/23b51b37b12b51a0947f687d1de4d3b5.svg
paper_authors.fullname: Chao Du
paper_authors.type: user
paper_authors.user: duchao
paper_authors._id: 67106d4b63f0a6c38b2633e0
paper_authors.name: Tianyu Pang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-20T15:05:04.503Z
paper_authors._id: 63d91b6d255ef6add20e1b38
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1675921369867-63d91b6d255ef6add20e1b38.jpeg
paper_authors.fullname: Tianyu Pang
paper_authors.type: user
paper_authors.user: P2333
paper_authors._id: 67106d4b63f0a6c38b2633e1
paper_authors.name: Zehan Wang
paper_authors._id: 67106d4b63f0a6c38b2633e2
paper_authors.name: Chongxuan Li
paper_authors._id: 67106d4b63f0a6c38b2633e3
paper_authors.name: Dong Xu
paper_title: Improving Long-Text Alignment for Text-to-Image Diffusion Models
paper_summary: The rapid advancement of text-to-image (T2I) diffusion models has enabled
  them to generate unprecedented results from given texts. However, as text
  inputs become longer, existing encoding methods like CLIP face limitations, and
  aligning the generated images with long texts becomes challenging. To tackle
  these issues, we propose LongAlign, which includes a segment-level encoding
  method for processing long texts and a decomposed preference optimization
  method for effective alignment training. For segment-level encoding, long texts
  are divided into multiple segments and processed separately. This method
  overcomes the maximum input length limits of pretrained encoding models. For
  preference optimization, we provide decomposed CLIP-based preference models to
  fine-tune diffusion models. Specifically, to utilize CLIP-based preference
  models for T2I alignment, we delve into their scoring mechanisms and find that
  the preference scores can be decomposed into two components: a text-relevant
  part that measures T2I alignment and a text-irrelevant part that assesses other
  visual aspects of human preference. Additionally, we find that the
  text-irrelevant part contributes to a common overfitting problem during
  fine-tuning. To address this, we propose a reweighting strategy that assigns
  different weights to these two components, thereby reducing overfitting and
  enhancing alignment. After fine-tuning 512 times 512 Stable Diffusion (SD)
  v1.5 for about 20 hours using our method, the fine-tuned SD outperforms
  stronger foundation models in T2I alignment, such as PixArt-alpha and
  Kandinsky v2.2. The code is available at
  https://github.com/luping-liu/LongAlign.
paper_ai_summary: LongAlign enhances text-to-image alignment for long texts by using segment-level encoding and decomposed preference optimization to fine-tune diffusion models.
paper_ai_keywords: text-to-image CLIP LongAlign segment-level encoding preference optimization CLIP-based preference models T2I alignment text-relevant part text-irrelevant part overfitting reweighting strategy Stable Diffusion