title: FRAP: Faithful and Realistic Text-to-Image Generation with Adaptive
    Prompt Weighting
summary: Text-to-image (T2I) diffusion models have demonstrated impressive
  capabilities in generating high-quality images given a text prompt. However,
  ensuring the prompt-image alignment remains a considerable challenge, i.e.,
  generating images that faithfully align with the prompt's semantics. Recent
  works attempt to improve the faithfulness by optimizing the latent code, which
  potentially could cause the latent code to go out-of-distribution and thus
  produce unrealistic images. In this paper, we propose FRAP, a simple, yet
  effective approach based on adaptively adjusting the per-token prompt weights
  to improve prompt-image alignment and authenticity of the generated images. We
  design an online algorithm to adaptively update each token's weight
  coefficient, which is achieved by minimizing a unified objective function that
  encourages object presence and the binding of object-modifier pairs. Through
  extensive evaluations, we show FRAP generates images with significantly higher
  prompt-image alignment to prompts from complex datasets, while having a lower
  average latency compared to recent latent code optimization methods, e.g., 4
  seconds faster than D&B on the COCO-Subject dataset. Furthermore, through
  visual comparisons and evaluation on the CLIP-IQA-Real metric, we show that
  FRAP not only improves prompt-image alignment but also generates more authentic
  images with realistic appearances. We also explore combining FRAP with prompt
  rewriting LLM to recover their degraded prompt-image alignment, where we
  observe improvements in both prompt-image alignment and image quality.
paper_authors._id: 66c73909a575572fcb8d3cf5
paper_authors.name: Liyao Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-22T19:50:14.732Z
paper_authors._id: 64ac49ccb7d86b40fd60a8dd
paper_authors.avatarUrl: /avatars/e9f5482cffdd1d5917523a496a3805f0.svg
paper_authors.fullname: Liyao Jiang
paper_authors.type: user
paper_authors.user: LiyaoJiang
paper_authors._id: 66c73909a575572fcb8d3cf6
paper_authors.name: Negar Hassanpour
paper_authors._id: 66c73909a575572fcb8d3cf7
paper_authors.name: Mohammad Salameh
paper_authors._id: 66c73909a575572fcb8d3cf8
paper_authors.name: Mohan Sai Singamsetti
paper_authors._id: 66c73909a575572fcb8d3cf9
paper_authors.name: Fengyu Sun
paper_authors._id: 66c73909a575572fcb8d3cfa
paper_authors.name: Wei Lu
paper_authors._id: 66c73909a575572fcb8d3cfb
paper_authors.name: Di Niu
paper_title: FRAP: Faithful and Realistic Text-to-Image Generation with Adaptive
    Prompt Weighting
paper_summary: Text-to-image (T2I) diffusion models have demonstrated impressive
  capabilities in generating high-quality images given a text prompt. However,
  ensuring the prompt-image alignment remains a considerable challenge, i.e.,
  generating images that faithfully align with the prompt's semantics. Recent
  works attempt to improve the faithfulness by optimizing the latent code, which
  potentially could cause the latent code to go out-of-distribution and thus
  produce unrealistic images. In this paper, we propose FRAP, a simple, yet
  effective approach based on adaptively adjusting the per-token prompt weights
  to improve prompt-image alignment and authenticity of the generated images. We
  design an online algorithm to adaptively update each token's weight
  coefficient, which is achieved by minimizing a unified objective function that
  encourages object presence and the binding of object-modifier pairs. Through
  extensive evaluations, we show FRAP generates images with significantly higher
  prompt-image alignment to prompts from complex datasets, while having a lower
  average latency compared to recent latent code optimization methods, e.g., 4
  seconds faster than D&B on the COCO-Subject dataset. Furthermore, through
  visual comparisons and evaluation on the CLIP-IQA-Real metric, we show that
  FRAP not only improves prompt-image alignment but also generates more authentic
  images with realistic appearances. We also explore combining FRAP with prompt
  rewriting LLM to recover their degraded prompt-image alignment, where we
  observe improvements in both prompt-image alignment and image quality.
paper_ai_keywords: diffusion models prompt-image alignment latent code per-token prompt weights online algorithm objective function object presence object-modifier pairs COCO-Subject dataset CLIP-IQA-Real metric prompt rewriting LLM