title: Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen
    Convolutional CLIP
summary: Open-vocabulary segmentation is a challenging task requiring segmenting and
  recognizing objects from an open set of categories. One way to address this
  challenge is to leverage multi-modal models, such as CLIP, to provide image and
  text features in a shared embedding space, which bridges the gap between
  closed-vocabulary and open-vocabulary recognition. Hence, existing methods
  often adopt a two-stage framework to tackle the problem, where the inputs first
  go through a mask generator and then through the CLIP model along with the
  predicted masks. This process involves extracting features from images multiple
  times, which can be ineffective and inefficient. By contrast, we propose to
  build everything into a single-stage framework using a shared Frozen
  Convolutional CLIP backbone, which not only significantly simplifies the
  current two-stage pipeline, but also remarkably yields a better accuracy-cost
  trade-off. The proposed FC-CLIP, benefits from the following observations: the
  frozen CLIP backbone maintains the ability of open-vocabulary classification
  and can also serve as a strong mask generator, and the convolutional CLIP
  generalizes well to a larger input resolution than the one used during
  contrastive image-text pretraining. When training on COCO panoptic data only
  and testing in a zero-shot manner, FC-CLIP achieve 26.8 PQ, 16.8 AP, and 34.1
  mIoU on ADE20K, 18.2 PQ, 27.9 mIoU on Mapillary Vistas, 44.0 PQ, 26.8 AP, 56.2
  mIoU on Cityscapes, outperforming the prior art by +4.2 PQ, +2.4 AP, +4.2 mIoU
  on ADE20K, +4.0 PQ on Mapillary Vistas and +20.1 PQ on Cityscapes,
  respectively. Additionally, the training and testing time of FC-CLIP is 7.5x
  and 6.6x significantly faster than the same prior art, while using 5.9x fewer
  parameters. FC-CLIP also sets a new state-of-the-art performance across various
  open-vocabulary semantic segmentation datasets. Code at
  https://github.com/bytedance/fc-clip
paper_authors._id: 64d06ed1e9cac0020baffdd3
paper_authors.name: Qihang Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-07T12:56:15.783Z
paper_authors._id: 648010b00b9d0f49849adb19
paper_authors.avatarUrl: /avatars/54419bb6e6ee788aba10cd64cd921204.svg
paper_authors.fullname: Qihang Yu
paper_authors.type: user
paper_authors.user: yucornetto
paper_authors._id: 64d06ed1e9cac0020baffdd4
paper_authors.name: Ju He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-04T09:37:02.316Z
paper_authors._id: 661c9059bcd78151e5c06ea1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/661c9059bcd78151e5c06ea1/27bfNo1LZeZQ77vWuAa10.png
paper_authors.fullname: Ju He
paper_authors.type: user
paper_authors.user: turkeyju
paper_authors._id: 64d06ed1e9cac0020baffdd5
paper_authors.name: Xueqing Deng
paper_authors._id: 64d06ed1e9cac0020baffdd6
paper_authors.name: Xiaohui Shen
paper_authors._id: 64d06ed1e9cac0020baffdd7
paper_authors.name: Liang-Chieh Chen
paper_title: Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen
    Convolutional CLIP
paper_summary: Open-vocabulary segmentation is a challenging task requiring segmenting and
  recognizing objects from an open set of categories. One way to address this
  challenge is to leverage multi-modal models, such as CLIP, to provide image and
  text features in a shared embedding space, which bridges the gap between
  closed-vocabulary and open-vocabulary recognition. Hence, existing methods
  often adopt a two-stage framework to tackle the problem, where the inputs first
  go through a mask generator and then through the CLIP model along with the
  predicted masks. This process involves extracting features from images multiple
  times, which can be ineffective and inefficient. By contrast, we propose to
  build everything into a single-stage framework using a shared Frozen
  Convolutional CLIP backbone, which not only significantly simplifies the
  current two-stage pipeline, but also remarkably yields a better accuracy-cost
  trade-off. The proposed FC-CLIP, benefits from the following observations: the
  frozen CLIP backbone maintains the ability of open-vocabulary classification
  and can also serve as a strong mask generator, and the convolutional CLIP
  generalizes well to a larger input resolution than the one used during
  contrastive image-text pretraining. When training on COCO panoptic data only
  and testing in a zero-shot manner, FC-CLIP achieve 26.8 PQ, 16.8 AP, and 34.1
  mIoU on ADE20K, 18.2 PQ, 27.9 mIoU on Mapillary Vistas, 44.0 PQ, 26.8 AP, 56.2
  mIoU on Cityscapes, outperforming the prior art by +4.2 PQ, +2.4 AP, +4.2 mIoU
  on ADE20K, +4.0 PQ on Mapillary Vistas and +20.1 PQ on Cityscapes,
  respectively. Additionally, the training and testing time of FC-CLIP is 7.5x
  and 6.6x significantly faster than the same prior art, while using 5.9x fewer
  parameters. FC-CLIP also sets a new state-of-the-art performance across various
  open-vocabulary semantic segmentation datasets. Code at
  https://github.com/bytedance/fc-clip
paper_ai_keywords: CLIP mask generator open-vocabulary segmentation multi-modal models shared embedding space Frozen Convolutional CLIP (FC-CLIP) contrastive image-text pretraining panoptic data zero-shot manner PQ (PQ likely means Panoptic Quality) AP (Average Precision) mIoU (mean Intersection over Union) COCO ADE20K Mapillary Vistas Cityscapes state-of-the-art performance open-vocabulary classification