Zero-shot spatial layout conditioning for text-to-image diffusion models
Large-scale text-to-image diffusion models have significantly improved the
state of the art in generative image modelling and allow for an intuitive and
powerful user interface to drive the image generation process. Expressing
spatial constraints, e.g. to position specific objects in particular locations,
is cumbersome using text; and current text-based image generation models are
not able to accurately follow such instructions. In this paper we consider
image generation from text associated with segments on the image canvas, which
combines an intuitive natural language interface with precise spatial control
over the generated content. We propose ZestGuide, a zero-shot segmentation
guidance approach that can be plugged into pre-trained text-to-image diffusion
models, and does not require any additional training. It leverages implicit
segmentation maps that can be extracted from cross-attention layers, and uses
them to align the generation with input masks. Our experimental results combine
high image quality with accurate alignment of generated content with input
segmentations, and improve over prior work both quantitatively and
qualitatively, including methods that require training on images with
corresponding segmentations. Compared to Paint with Words, the previous
state-of-the art in image generation with zero-shot segmentation conditioning,
we improve by 5 to 10 mIoU points on the COCO dataset with similar FID scores.
649a5ce774485a2285aa7260
Guillaume Couairon
claimed_verified
2023-06-28T20:14:49.126Z
6304b88ebad6ce7fc026c0b0
https://cdn-avatars.huggingface.co/v1/production/uploads/6304b88ebad6ce7fc026c0b0/OypG9j7KLbhdtgNCTAcnI.jpeg
Guillaume Couairon
user
gcouairon
649a5ce774485a2285aa7261
Marlène Careil
claimed_verified
2024-04-02T11:54:53.409Z
630620e1df993a789e644129
/avatars/315c41ecad079ae57350b7662382b2cb.svg
Careil
user
Marl
649a5ce774485a2285aa7262
Matthieu Cord
649a5ce774485a2285aa7263
Stéphane Lathuilière
649a5ce774485a2285aa7264
Jakob Verbeek
Zero-shot spatial layout conditioning for text-to-image diffusion models
Large-scale text-to-image diffusion models have significantly improved the
state of the art in generative image modelling and allow for an intuitive and
powerful user interface to drive the image generation process. Expressing
spatial constraints, e.g. to position specific objects in particular locations,
is cumbersome using text; and current text-based image generation models are
not able to accurately follow such instructions. In this paper we consider
image generation from text associated with segments on the image canvas, which
combines an intuitive natural language interface with precise spatial control
over the generated content. We propose ZestGuide, a zero-shot segmentation
guidance approach that can be plugged into pre-trained text-to-image diffusion
models, and does not require any additional training. It leverages implicit
segmentation maps that can be extracted from cross-attention layers, and uses
them to align the generation with input masks. Our experimental results combine
high image quality with accurate alignment of generated content with input
segmentations, and improve over prior work both quantitatively and
qualitatively, including methods that require training on images with
corresponding segmentations. Compared to Paint with Words, the previous
state-of-the art in image generation with zero-shot segmentation conditioning,
we improve by 5 to 10 mIoU points on the COCO dataset with similar FID scores.
ZestGuide enhances pre-trained text-to-image diffusion models with zero-shot segmentation guidance using implicit segmentation maps from cross-attention layers, improving image quality and alignment accuracy.
diffusion models text-to-image segmentation cross-attention layers implicit segmentation maps zero-shot segmentation guidance ZestGuide COCO dataset mIoU FID scores