title: Diffusion Models for Zero-Shot Open-Vocabulary Segmentation
summary: The variety of objects in the real world is nearly unlimited and is thus
  impossible to capture using models trained on a fixed set of categories. As a
  result, in recent years, open-vocabulary methods have attracted the interest of
  the community. This paper proposes a new method for zero-shot open-vocabulary
  segmentation. Prior work largely relies on contrastive training using
  image-text pairs, leveraging grouping mechanisms to learn image features that
  are both aligned with language and well-localised. This however can introduce
  ambiguity as the visual appearance of images with similar captions often
  varies. Instead, we leverage the generative properties of large-scale
  text-to-image diffusion models to sample a set of support images for a given
  textual category. This provides a distribution of appearances for a given text
  circumventing the ambiguity problem. We further propose a mechanism that
  considers the contextual background of the sampled images to better localise
  objects and segment the background directly. We show that our method can be
  used to ground several existing pre-trained self-supervised feature extractors
  in natural language and provide explainable predictions by mapping back to
  regions in the support set. Our proposal is training-free, relying on
  pre-trained components only, yet, shows strong performance on a range of
  open-vocabulary segmentation benchmarks, obtaining a lead of more than 10% on
  the Pascal VOC benchmark.
paper_authors._id: 648bfc021c113a0526221dfc
paper_authors.name: Laurynas Karazija
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-19T07:15:52.907Z
paper_authors._id: 6394e1c20663665c5f019e31
paper_authors.avatarUrl: /avatars/85840faf1055ec2fcae495db6f042a27.svg
paper_authors.fullname: Laurynas
paper_authors.type: user
paper_authors.user: lkarazija
paper_authors._id: 648bfc021c113a0526221dfd
paper_authors.name: Iro Laina
paper_authors._id: 648bfc021c113a0526221dfe
paper_authors.name: Andrea Vedaldi
paper_authors._id: 648bfc021c113a0526221dff
paper_authors.name: Christian Rupprecht
paper_title: Diffusion Models for Zero-Shot Open-Vocabulary Segmentation
paper_summary: The variety of objects in the real world is nearly unlimited and is thus
  impossible to capture using models trained on a fixed set of categories. As a
  result, in recent years, open-vocabulary methods have attracted the interest of
  the community. This paper proposes a new method for zero-shot open-vocabulary
  segmentation. Prior work largely relies on contrastive training using
  image-text pairs, leveraging grouping mechanisms to learn image features that
  are both aligned with language and well-localised. This however can introduce
  ambiguity as the visual appearance of images with similar captions often
  varies. Instead, we leverage the generative properties of large-scale
  text-to-image diffusion models to sample a set of support images for a given
  textual category. This provides a distribution of appearances for a given text
  circumventing the ambiguity problem. We further propose a mechanism that
  considers the contextual background of the sampled images to better localise
  objects and segment the background directly. We show that our method can be
  used to ground several existing pre-trained self-supervised feature extractors
  in natural language and provide explainable predictions by mapping back to
  regions in the support set. Our proposal is training-free, relying on
  pre-trained components only, yet, shows strong performance on a range of
  open-vocabulary segmentation benchmarks, obtaining a lead of more than 10% on
  the Pascal VOC benchmark.
paper_ai_summary: Zero-shot open-vocabulary segmentation uses text-to-image diffusion models to sample support images, enhancing localization and background segmentation with pre-trained feature extractors.
paper_ai_keywords: zero-shot open-vocabulary segmentation contrastive training image-text pairs text-to-image diffusion models support images contextual background pre-trained feature extractors Pascal VOC benchmark