title: Grounded Text-to-Image Synthesis with Attention Refocusing
summary: Driven by scalable diffusion models trained on large-scale paired text-image
  datasets, text-to-image synthesis methods have shown compelling results.
  However, these models still fail to precisely follow the text prompt when
  multiple objects, attributes, and spatial compositions are involved in the
  prompt. In this paper, we identify the potential reasons in both the
  cross-attention and self-attention layers of the diffusion model. We propose
  two novel losses to refocus the attention maps according to a given layout
  during the sampling process. We perform comprehensive experiments on the
  DrawBench and HRS benchmarks using layouts synthesized by Large Language
  Models, showing that our proposed losses can be integrated easily and
  effectively into existing text-to-image methods and consistently improve their
  alignment between the generated images and the text prompts.
paper_authors._id: 6482a54fd3973485034deac0
paper_authors.name: Quynh Phung
paper_authors._id: 6482a54fd3973485034deac1
paper_authors.name: Songwei Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:30:53.093Z
paper_authors._id: 631cdc696e623740a78fd864
paper_authors.avatarUrl: /avatars/f610873f6080453b655fb230d6f3b765.svg
paper_authors.fullname: Songwei Ge
paper_authors.type: user
paper_authors.user: songweig
paper_authors._id: 6482a54fd3973485034deac2
paper_authors.name: Jia-Bin Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:31:15.328Z
paper_authors._id: 641c139b73296f7ee256970c
paper_authors.avatarUrl: /avatars/5a2550d95e686640242840ad3bd0e680.svg
paper_authors.fullname: Jiabin Huang
paper_authors.type: user
paper_authors.user: YellowAddice
paper_title: Grounded Text-to-Image Synthesis with Attention Refocusing
paper_summary: Driven by scalable diffusion models trained on large-scale paired text-image
  datasets, text-to-image synthesis methods have shown compelling results.
  However, these models still fail to precisely follow the text prompt when
  multiple objects, attributes, and spatial compositions are involved in the
  prompt. In this paper, we identify the potential reasons in both the
  cross-attention and self-attention layers of the diffusion model. We propose
  two novel losses to refocus the attention maps according to a given layout
  during the sampling process. We perform comprehensive experiments on the
  DrawBench and HRS benchmarks using layouts synthesized by Large Language
  Models, showing that our proposed losses can be integrated easily and
  effectively into existing text-to-image methods and consistently improve their
  alignment between the generated images and the text prompts.
paper_ai_keywords: diffusion models paired attention cross-attention self-specialization hierarchical losses attention maps sampling.Encrionichi layout sampling process DrawBench HRS benchmarks Large Language Models alignment generated images text prompts