Compositional Text-to-Image Generation with Dense Blob Representations
Existing text-to-image models struggle to follow complex text prompts,
raising the need for extra grounding inputs for better controllability. In this
work, we propose to decompose a scene into visual primitives - denoted as dense
blob representations - that contain fine-grained details of the scene while
being modular, human-interpretable, and easy-to-construct. Based on blob
representations, we develop a blob-grounded text-to-image diffusion model,
termed BlobGEN, for compositional generation. Particularly, we introduce a new
masked cross-attention module to disentangle the fusion between blob
representations and visual features. To leverage the compositionality of large
language models (LLMs), we introduce a new in-context learning approach to
generate blob representations from text prompts. Our extensive experiments show
that BlobGEN achieves superior zero-shot generation quality and better
layout-guided controllability on MS-COCO. When augmented by LLMs, our method
exhibits superior numerical and spatial correctness on compositional image
generation benchmarks. Project page: https://blobgen-2d.github.io.
6644c6eb3744e4bcd5b7a53b
Weili Nie
admin_assigned
2024-05-15T17:16:21.819Z
64c1a69e226e016da8450ae2
/avatars/54c161e8b8543244ed13cbe47017624e.svg
Weili Nie
user
xiaoli08
6644c6eb3744e4bcd5b7a53c
Sifei Liu
admin_assigned
2024-05-15T17:22:29.311Z
62fab69f8cd542e895bafd6e
/avatars/c553bff4bd52b9a4f79e9c76fa22e27e.svg
Sifei Liu
user
zwrq
6644c6eb3744e4bcd5b7a53d
Morteza Mardani
admin_assigned
2024-05-15T17:22:36.597Z
6350842a0f376d3c482bc04a
/avatars/dca5f6b3e867ed28f80882ea2b32c2c8.svg
morteza mardani
user
mortezamardani
6644c6eb3744e4bcd5b7a53e
Chao Liu
6644c6eb3744e4bcd5b7a53f
Benjamin Eckart
6644c6eb3744e4bcd5b7a540
Arash Vahdat
admin_assigned
2024-05-15T17:23:02.125Z
62f6956cd3bdacb7eec02920
/avatars/b22db0823311f866c00db2efc4b9f814.svg
Arash Vahdat
user
avahdat
Compositional Text-to-Image Generation with Dense Blob Representations
Existing text-to-image models struggle to follow complex text prompts,
raising the need for extra grounding inputs for better controllability. In this
work, we propose to decompose a scene into visual primitives - denoted as dense
blob representations - that contain fine-grained details of the scene while
being modular, human-interpretable, and easy-to-construct. Based on blob
representations, we develop a blob-grounded text-to-image diffusion model,
termed BlobGEN, for compositional generation. Particularly, we introduce a new
masked cross-attention module to disentangle the fusion between blob
representations and visual features. To leverage the compositionality of large
language models (LLMs), we introduce a new in-context learning approach to
generate blob representations from text prompts. Our extensive experiments show
that BlobGEN achieves superior zero-shot generation quality and better
layout-guided controllability on MS-COCO. When augmented by LLMs, our method
exhibits superior numerical and spatial correctness on compositional image
generation benchmarks. Project page: https://blobgen-2d.github.io.
BlobGEN, a blob-grounded text-to-image diffusion model with masked cross-attention and in-context learning from LLMs, excels in zero-shot generation and layout control, outperforming benchmarks in numerical and spatial correctness.
denoising autoencoders diffusion models blob representations masked cross-attention large language models in-context learning MS-COCO compositional image generation