title: Zero-shot spatial layout conditioning for text-to-image diffusion models
summary: Large-scale text-to-image diffusion models have significantly improved the
  state of the art in generative image modelling and allow for an intuitive and
  powerful user interface to drive the image generation process. Expressing
  spatial constraints, e.g. to position specific objects in particular locations,
  is cumbersome using text; and current text-based image generation models are
  not able to accurately follow such instructions. In this paper we consider
  image generation from text associated with segments on the image canvas, which
  combines an intuitive natural language interface with precise spatial control
  over the generated content. We propose ZestGuide, a zero-shot segmentation
  guidance approach that can be plugged into pre-trained text-to-image diffusion
  models, and does not require any additional training. It leverages implicit
  segmentation maps that can be extracted from cross-attention layers, and uses
  them to align the generation with input masks. Our experimental results combine
  high image quality with accurate alignment of generated content with input
  segmentations, and improve over prior work both quantitatively and
  qualitatively, including methods that require training on images with
  corresponding segmentations. Compared to Paint with Words, the previous
  state-of-the art in image generation with zero-shot segmentation conditioning,
  we improve by 5 to 10 mIoU points on the COCO dataset with similar FID scores.
paper_authors._id: 649a5ce774485a2285aa7260
paper_authors.name: Guillaume Couairon
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-28T20:14:49.126Z
paper_authors._id: 6304b88ebad6ce7fc026c0b0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6304b88ebad6ce7fc026c0b0/OypG9j7KLbhdtgNCTAcnI.jpeg
paper_authors.fullname: Guillaume Couairon
paper_authors.type: user
paper_authors.user: gcouairon
paper_authors._id: 649a5ce774485a2285aa7261
paper_authors.name: Marlène Careil
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-02T11:54:53.409Z
paper_authors._id: 630620e1df993a789e644129
paper_authors.avatarUrl: /avatars/315c41ecad079ae57350b7662382b2cb.svg
paper_authors.fullname: Careil
paper_authors.type: user
paper_authors.user: Marl
paper_authors._id: 649a5ce774485a2285aa7262
paper_authors.name: Matthieu Cord
paper_authors._id: 649a5ce774485a2285aa7263
paper_authors.name: Stéphane Lathuilière
paper_authors._id: 649a5ce774485a2285aa7264
paper_authors.name: Jakob Verbeek
paper_title: Zero-shot spatial layout conditioning for text-to-image diffusion models
paper_summary: Large-scale text-to-image diffusion models have significantly improved the
  state of the art in generative image modelling and allow for an intuitive and
  powerful user interface to drive the image generation process. Expressing
  spatial constraints, e.g. to position specific objects in particular locations,
  is cumbersome using text; and current text-based image generation models are
  not able to accurately follow such instructions. In this paper we consider
  image generation from text associated with segments on the image canvas, which
  combines an intuitive natural language interface with precise spatial control
  over the generated content. We propose ZestGuide, a zero-shot segmentation
  guidance approach that can be plugged into pre-trained text-to-image diffusion
  models, and does not require any additional training. It leverages implicit
  segmentation maps that can be extracted from cross-attention layers, and uses
  them to align the generation with input masks. Our experimental results combine
  high image quality with accurate alignment of generated content with input
  segmentations, and improve over prior work both quantitatively and
  qualitatively, including methods that require training on images with
  corresponding segmentations. Compared to Paint with Words, the previous
  state-of-the art in image generation with zero-shot segmentation conditioning,
  we improve by 5 to 10 mIoU points on the COCO dataset with similar FID scores.
paper_ai_keywords: diffusion models text-to-image generative image modelling user interface spatial constraints natural language interface segmentation guidance ZestGuide pre-trained implicit segmentation maps cross-attention layers input masks image quality alignment generated content zero-shot segmentation conditioning Paint with Words COCO dataset mIoU FID scores