title: gen2seg: Generative Models Enable Generalizable Instance Segmentation
summary: By pretraining to synthesize coherent images from perturbed inputs,
  generative models inherently learn to understand object boundaries and scene
  compositions. How can we repurpose these generative representations for
  general-purpose perceptual organization? We finetune Stable Diffusion and MAE
  (encoder+decoder) for category-agnostic instance segmentation using our
  instance coloring loss exclusively on a narrow set of object types (indoor
  furnishings and cars). Surprisingly, our models exhibit strong zero-shot
  generalization, accurately segmenting objects of types and styles unseen in
  finetuning (and in many cases, MAE's ImageNet-1K pretraining too). Our
  best-performing models closely approach the heavily supervised SAM when
  evaluated on unseen object types and styles, and outperform it when segmenting
  fine structures and ambiguous boundaries. In contrast, existing promptable
  segmentation architectures or discriminatively pretrained models fail to
  generalize. This suggests that generative models learn an inherent grouping
  mechanism that transfers across categories and domains, even without
  internet-scale pretraining. Code, pretrained models, and demos are available on
  our website.
paper_authors._id: 682e9007841407673735967d
paper_authors.name: Om Khangaonkar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-22T07:16:17.333Z
paper_authors._id: 680873cdcdb08bc332e3145a
paper_authors.avatarUrl: /avatars/b109d7cdaf0a355262c84ec54d1ccd52.svg
paper_authors.fullname: Om Khangaonkar
paper_authors.type: user
paper_authors.user: reachomk
paper_authors._id: 682e9007841407673735967e
paper_authors.name: Hamed Pirsiavash
paper_title: gen2seg: Generative Models Enable Generalizable Instance Segmentation
paper_summary: By pretraining to synthesize coherent images from perturbed inputs,
  generative models inherently learn to understand object boundaries and scene
  compositions. How can we repurpose these generative representations for
  general-purpose perceptual organization? We finetune Stable Diffusion and MAE
  (encoder+decoder) for category-agnostic instance segmentation using our
  instance coloring loss exclusively on a narrow set of object types (indoor
  furnishings and cars). Surprisingly, our models exhibit strong zero-shot
  generalization, accurately segmenting objects of types and styles unseen in
  finetuning (and in many cases, MAE's ImageNet-1K pretraining too). Our
  best-performing models closely approach the heavily supervised SAM when
  evaluated on unseen object types and styles, and outperform it when segmenting
  fine structures and ambiguous boundaries. In contrast, existing promptable
  segmentation architectures or discriminatively pretrained models fail to
  generalize. This suggests that generative models learn an inherent grouping
  mechanism that transfers across categories and domains, even without
  internet-scale pretraining. Code, pretrained models, and demos are available on
  our website.
paper_ai_summary: Generative models fine-tuned for instance segmentation demonstrate strong zero-shot performance on unseen objects and styles, surpassing discriminatively pretrained models.
paper_ai_keywords: generative models Stable Diffusion MAE instance segmentation instance coloring loss zero-shot generalization SAM Internet-scale pretraining