title: Unsupervised Universal Image Segmentation
summary: Several unsupervised image segmentation approaches have been proposed which
  eliminate the need for dense manually-annotated segmentation masks; current
  models separately handle either semantic segmentation (e.g., STEGO) or
  class-agnostic instance segmentation (e.g., CutLER), but not both (i.e.,
  panoptic segmentation). We propose an Unsupervised Universal Segmentation model
  (U2Seg) adept at performing various image segmentation tasks -- instance,
  semantic and panoptic -- using a novel unified framework. U2Seg generates
  pseudo semantic labels for these segmentation tasks via leveraging
  self-supervised models followed by clustering; each cluster represents
  different semantic and/or instance membership of pixels. We then self-train the
  model on these pseudo semantic labels, yielding substantial performance gains
  over specialized methods tailored to each task: a +2.6 AP^{box} boost
  vs. CutLER in unsupervised instance segmentation on COCO and a +7.0 PixelAcc
  increase (vs. STEGO) in unsupervised semantic segmentation on COCOStuff.
  Moreover, our method sets up a new baseline for unsupervised panoptic
  segmentation, which has not been previously explored. U2Seg is also a strong
  pretrained model for few-shot segmentation, surpassing CutLER by +5.0
  AP^{mask} when trained on a low-data regime, e.g., only 1% COCO
  labels. We hope our simple yet effective method can inspire more research on
  unsupervised universal image segmentation.
paper_authors._id: 658e4f96f5a209eeacb4404c
paper_authors.name: Dantong Niu
paper_authors._id: 658e4f96f5a209eeacb4404d
paper_authors.name: Xudong Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-21T08:30:54.754Z
paper_authors._id: 63d6f76f44f1d8fbe58fbd9d
paper_authors.avatarUrl: /avatars/184f248a090b543e576a97b7f46ab3ef.svg
paper_authors.fullname: Xudong Wang
paper_authors.type: user
paper_authors.user: xudongw
paper_authors._id: 658e4f96f5a209eeacb4404e
paper_authors.name: Xinyang Han
paper_authors._id: 658e4f96f5a209eeacb4404f
paper_authors.name: Long Lian
paper_authors._id: 658e4f96f5a209eeacb44050
paper_authors.name: Roei Herzig
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-27T09:50:01.072Z
paper_authors._id: 667c5764186b27ef806636d3
paper_authors.avatarUrl: /avatars/5c08f0109bc0e350624112c0aff544f6.svg
paper_authors.fullname: Roei Herzig
paper_authors.type: user
paper_authors.user: roeiherz
paper_authors._id: 658e4f96f5a209eeacb44051
paper_authors.name: Trevor Darrell
paper_title: Unsupervised Universal Image Segmentation
paper_summary: Several unsupervised image segmentation approaches have been proposed which
  eliminate the need for dense manually-annotated segmentation masks; current
  models separately handle either semantic segmentation (e.g., STEGO) or
  class-agnostic instance segmentation (e.g., CutLER), but not both (i.e.,
  panoptic segmentation). We propose an Unsupervised Universal Segmentation model
  (U2Seg) adept at performing various image segmentation tasks -- instance,
  semantic and panoptic -- using a novel unified framework. U2Seg generates
  pseudo semantic labels for these segmentation tasks via leveraging
  self-supervised models followed by clustering; each cluster represents
  different semantic and/or instance membership of pixels. We then self-train the
  model on these pseudo semantic labels, yielding substantial performance gains
  over specialized methods tailored to each task: a +2.6 AP^{box} boost
  vs. CutLER in unsupervised instance segmentation on COCO and a +7.0 PixelAcc
  increase (vs. STEGO) in unsupervised semantic segmentation on COCOStuff.
  Moreover, our method sets up a new baseline for unsupervised panoptic
  segmentation, which has not been previously explored. U2Seg is also a strong
  pretrained model for few-shot segmentation, surpassing CutLER by +5.0
  AP^{mask} when trained on a low-data regime, e.g., only 1% COCO
  labels. We hope our simple yet effective method can inspire more research on
  unsupervised universal image segmentation.
paper_ai_keywords: unsupervised image segmentation dense manually-annotated segmentation masks semantic segmentation STEGO class-agnostic instance segmentation CutLER panoptic segmentation Unsupervised Universal Segmentation model U2Seg pseudo semantic labels self-supervised models clustering self-train AP$^{\text{box}}$ COCO PixelAcc COCOStuff unsupervised panoptic segmentation few-shot segmentation AP$^{\text{mask}}$