title: Compositional Text-to-Image Generation with Dense Blob Representations
summary: Existing text-to-image models struggle to follow complex text prompts,
  raising the need for extra grounding inputs for better controllability. In this
  work, we propose to decompose a scene into visual primitives - denoted as dense
  blob representations - that contain fine-grained details of the scene while
  being modular, human-interpretable, and easy-to-construct. Based on blob
  representations, we develop a blob-grounded text-to-image diffusion model,
  termed BlobGEN, for compositional generation. Particularly, we introduce a new
  masked cross-attention module to disentangle the fusion between blob
  representations and visual features. To leverage the compositionality of large
  language models (LLMs), we introduce a new in-context learning approach to
  generate blob representations from text prompts. Our extensive experiments show
  that BlobGEN achieves superior zero-shot generation quality and better
  layout-guided controllability on MS-COCO. When augmented by LLMs, our method
  exhibits superior numerical and spatial correctness on compositional image
  generation benchmarks. Project page: https://blobgen-2d.github.io.
paper_authors._id: 6644c6eb3744e4bcd5b7a53b
paper_authors.name: Weili Nie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-15T17:16:21.819Z
paper_authors._id: 64c1a69e226e016da8450ae2
paper_authors.avatarUrl: /avatars/54c161e8b8543244ed13cbe47017624e.svg
paper_authors.fullname: Weili Nie
paper_authors.type: user
paper_authors.user: xiaoli08
paper_authors._id: 6644c6eb3744e4bcd5b7a53c
paper_authors.name: Sifei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-15T17:22:29.311Z
paper_authors._id: 62fab69f8cd542e895bafd6e
paper_authors.avatarUrl: /avatars/c553bff4bd52b9a4f79e9c76fa22e27e.svg
paper_authors.fullname: Sifei Liu
paper_authors.type: user
paper_authors.user: zwrq
paper_authors._id: 6644c6eb3744e4bcd5b7a53d
paper_authors.name: Morteza Mardani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-15T17:22:36.597Z
paper_authors._id: 6350842a0f376d3c482bc04a
paper_authors.avatarUrl: /avatars/dca5f6b3e867ed28f80882ea2b32c2c8.svg
paper_authors.fullname: morteza mardani
paper_authors.type: user
paper_authors.user: mortezamardani
paper_authors._id: 6644c6eb3744e4bcd5b7a53e
paper_authors.name: Chao Liu
paper_authors._id: 6644c6eb3744e4bcd5b7a53f
paper_authors.name: Benjamin Eckart
paper_authors._id: 6644c6eb3744e4bcd5b7a540
paper_authors.name: Arash Vahdat
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-15T17:23:02.125Z
paper_authors._id: 62f6956cd3bdacb7eec02920
paper_authors.avatarUrl: /avatars/b22db0823311f866c00db2efc4b9f814.svg
paper_authors.fullname: Arash Vahdat
paper_authors.type: user
paper_authors.user: avahdat
paper_title: Compositional Text-to-Image Generation with Dense Blob Representations
paper_summary: Existing text-to-image models struggle to follow complex text prompts,
  raising the need for extra grounding inputs for better controllability. In this
  work, we propose to decompose a scene into visual primitives - denoted as dense
  blob representations - that contain fine-grained details of the scene while
  being modular, human-interpretable, and easy-to-construct. Based on blob
  representations, we develop a blob-grounded text-to-image diffusion model,
  termed BlobGEN, for compositional generation. Particularly, we introduce a new
  masked cross-attention module to disentangle the fusion between blob
  representations and visual features. To leverage the compositionality of large
  language models (LLMs), we introduce a new in-context learning approach to
  generate blob representations from text prompts. Our extensive experiments show
  that BlobGEN achieves superior zero-shot generation quality and better
  layout-guided controllability on MS-COCO. When augmented by LLMs, our method
  exhibits superior numerical and spatial correctness on compositional image
  generation benchmarks. Project page: https://blobgen-2d.github.io.
paper_ai_keywords: dense blob representations scene decomposition visual primitives blob-grounded text-to-image diffusion model BlobGEN masked cross-attention compositionality large language models (LLMs) in-context learning zero-shot generation quality layout-guided controllability compositional image generation benchmarks numerical and spatial correctness