title: FrozenSeg: Harmonizing Frozen Foundation Models for Open-Vocabulary
    Segmentation
summary: Open-vocabulary segmentation poses significant challenges, as it requires
  segmenting and recognizing objects across an open set of categories in
  unconstrained environments. Building on the success of powerful vision-language
  (ViL) foundation models, such as CLIP, recent efforts sought to harness their
  zero-short capabilities to recognize unseen categories. Despite notable
  performance improvements, these models still encounter the critical issue of
  generating precise mask proposals for unseen categories and scenarios,
  resulting in inferior segmentation performance eventually. To address this
  challenge, we introduce a novel approach, FrozenSeg, designed to integrate
  spatial knowledge from a localization foundation model (e.g., SAM) and semantic
  knowledge extracted from a ViL model (e.g., CLIP), in a synergistic framework.
  Taking the ViL model's visual encoder as the feature backbone, we inject the
  space-aware feature into the learnable queries and CLIP features within the
  transformer decoder. In addition, we devise a mask proposal ensemble strategy
  for further improving the recall rate and mask quality. To fully exploit
  pre-trained knowledge while minimizing training overhead, we freeze both
  foundation models, focusing optimization efforts solely on a lightweight
  transformer decoder for mask proposal generation-the performance bottleneck.
  Extensive experiments demonstrate that FrozenSeg advances state-of-the-art
  results across various segmentation benchmarks, trained exclusively on COCO
  panoptic data, and tested in a zero-shot manner. Code is available at
  https://github.com/chenxi52/FrozenSeg.
paper_authors._id: 66dad96e75b3337a858eed29
paper_authors.name: Xi Chen
paper_authors._id: 66dad96e75b3337a858eed2a
paper_authors.name: Haosen Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-06T11:15:13.600Z
paper_authors._id: 64c281e30344289c376c0fa4
paper_authors.avatarUrl: /avatars/dae01f7054da8c9517e3808a6f10eb1a.svg
paper_authors.fullname: HY
paper_authors.type: user
paper_authors.user: happy0612
paper_authors._id: 66dad96e75b3337a858eed2b
paper_authors.name: Sheng Jin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-06T11:15:39.300Z
paper_authors._id: 662e3d4e77909dd769cdc53a
paper_authors.avatarUrl: /avatars/ffc32ff17aad415eef7eff6d56fe3f75.svg
paper_authors.fullname: ShengJin
paper_authors.type: user
paper_authors.user: ShengJin
paper_authors._id: 66dad96e75b3337a858eed2c
paper_authors.name: Xiatian Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-06T11:15:28.374Z
paper_authors._id: 647ef9e7aa8c04bbf9362a8a
paper_authors.avatarUrl: /avatars/dc23e15acf7ee7f3b9f1a68c2716a66d.svg
paper_authors.fullname: Xiatian Zhu
paper_authors.type: user
paper_authors.user: Xiatian-Zhu
paper_authors._id: 66dad96e75b3337a858eed2d
paper_authors.name: Hongxun Yao
paper_title: FrozenSeg: Harmonizing Frozen Foundation Models for Open-Vocabulary
    Segmentation
paper_summary: Open-vocabulary segmentation poses significant challenges, as it requires
  segmenting and recognizing objects across an open set of categories in
  unconstrained environments. Building on the success of powerful vision-language
  (ViL) foundation models, such as CLIP, recent efforts sought to harness their
  zero-short capabilities to recognize unseen categories. Despite notable
  performance improvements, these models still encounter the critical issue of
  generating precise mask proposals for unseen categories and scenarios,
  resulting in inferior segmentation performance eventually. To address this
  challenge, we introduce a novel approach, FrozenSeg, designed to integrate
  spatial knowledge from a localization foundation model (e.g., SAM) and semantic
  knowledge extracted from a ViL model (e.g., CLIP), in a synergistic framework.
  Taking the ViL model's visual encoder as the feature backbone, we inject the
  space-aware feature into the learnable queries and CLIP features within the
  transformer decoder. In addition, we devise a mask proposal ensemble strategy
  for further improving the recall rate and mask quality. To fully exploit
  pre-trained knowledge while minimizing training overhead, we freeze both
  foundation models, focusing optimization efforts solely on a lightweight
  transformer decoder for mask proposal generation-the performance bottleneck.
  Extensive experiments demonstrate that FrozenSeg advances state-of-the-art
  results across various segmentation benchmarks, trained exclusively on COCO
  panoptic data, and tested in a zero-shot manner. Code is available at
  https://github.com/chenxi52/FrozenSeg.
paper_ai_keywords: open-vocabulary segmentation zero-shot capabilities vision-language (ViL) foundation models CLIP spatial knowledge localization foundation model (SAM) semantic knowledge visual encoder learnable queries transformer decoder space-aware feature mask proposal ensemble strategy recall rate mask quality pre-trained knowledge training overhead COCO panoptic data