title: SLiMe: Segment Like Me
summary: Significant strides have been made using large vision-language models, like
  Stable Diffusion (SD), for a variety of downstream tasks, including image
  editing, image correspondence, and 3D shape generation. Inspired by these
  advancements, we explore leveraging these extensive vision-language models for
  segmenting images at any desired granularity using as few as one annotated
  sample by proposing SLiMe. SLiMe frames this problem as an optimization task.
  Specifically, given a single training image and its segmentation mask, we first
  extract attention maps, including our novel "weighted accumulated
  self-attention map" from the SD prior. Then, using the extracted attention
  maps, the text embeddings of Stable Diffusion are optimized such that, each of
  them, learn about a single segmented region from the training image. These
  learned embeddings then highlight the segmented region in the attention maps,
  which in turn can then be used to derive the segmentation map. This enables
  SLiMe to segment any real-world image during inference with the granularity of
  the segmented region in the training image, using just one example. Moreover,
  leveraging additional training data when available, i.e. few-shot, improves the
  performance of SLiMe. We carried out a knowledge-rich set of experiments
  examining various design factors and showed that SLiMe outperforms other
  existing one-shot and few-shot segmentation methods.
paper_authors._id: 64f9321fb2c85cf9e521c01c
paper_authors.name: Aliasghar Khani
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-07T06:55:42.566Z
paper_authors._id: 6372b048f7e99092f9066683
paper_authors.avatarUrl: /avatars/e78eb302089da7e5073f9a8f4118e2d7.svg
paper_authors.fullname: aliasghar
paper_authors.type: user
paper_authors.user: aka225
paper_authors._id: 64f9321fb2c85cf9e521c01d
paper_authors.name: Saeid Asgari Taghanaki
paper_authors._id: 64f9321fb2c85cf9e521c01e
paper_authors.name: Aditya Sanghi
paper_authors._id: 64f9321fb2c85cf9e521c01f
paper_authors.name: Ali Mahdavi Amiri
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-07T12:54:41.364Z
paper_authors._id: 64233145d654afd693193a17
paper_authors.avatarUrl: /avatars/9c668059e68773fab120f1a12c82f3ed.svg
paper_authors.fullname: Ali Mahdavi-Amiri
paper_authors.type: user
paper_authors.user: amahdavi
paper_authors._id: 64f9321fb2c85cf9e521c020
paper_authors.name: Ghassan Hamarneh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-07T13:23:37.831Z
paper_authors._id: 64f9cc8ad56420ff571ab35c
paper_authors.avatarUrl: /avatars/87085c7f4ca3a0c30b64c4486b89353c.svg
paper_authors.fullname: Ghassan Hamarneh
paper_authors.type: user
paper_authors.user: Hamarneh
paper_title: SLiMe: Segment Like Me
paper_summary: Significant strides have been made using large vision-language models, like
  Stable Diffusion (SD), for a variety of downstream tasks, including image
  editing, image correspondence, and 3D shape generation. Inspired by these
  advancements, we explore leveraging these extensive vision-language models for
  segmenting images at any desired granularity using as few as one annotated
  sample by proposing SLiMe. SLiMe frames this problem as an optimization task.
  Specifically, given a single training image and its segmentation mask, we first
  extract attention maps, including our novel "weighted accumulated
  self-attention map" from the SD prior. Then, using the extracted attention
  maps, the text embeddings of Stable Diffusion are optimized such that, each of
  them, learn about a single segmented region from the training image. These
  learned embeddings then highlight the segmented region in the attention maps,
  which in turn can then be used to derive the segmentation map. This enables
  SLiMe to segment any real-world image during inference with the granularity of
  the segmented region in the training image, using just one example. Moreover,
  leveraging additional training data when available, i.e. few-shot, improves the
  performance of SLiMe. We carried out a knowledge-rich set of experiments
  examining various design factors and showed that SLiMe outperforms other
  existing one-shot and few-shot segmentation methods.
paper_ai_summary: SLiMe segments images at desired granularity using Stable Diffusion with minimal annotations and outperforms existing one-shot and few-shot segmentation methods.
paper_ai_keywords: Stable Diffusion attention maps weighted accumulated self-attention map text embeddings optimization task one-shot segmentation few-shot segmentation