title: CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor
summary: Existing open-vocabulary image segmentation methods require a fine-tuning
  step on mask annotations and/or image-text datasets. Mask labels are
  labor-intensive, which limits the number of categories in segmentation
  datasets. As a result, the open-vocabulary capacity of pre-trained VLMs is
  severely reduced after fine-tuning. However, without fine-tuning, VLMs trained
  under weak image-text supervision tend to make suboptimal mask predictions when
  there are text queries referring to non-existing concepts in the image. To
  alleviate these issues, we introduce a novel recurrent framework that
  progressively filters out irrelevant texts and enhances mask quality without
  training efforts. The recurrent unit is a two-stage segmenter built upon a VLM
  with frozen weights. Thus, our model retains the VLM's broad vocabulary space
  and strengthens its segmentation capability. Experimental results show that our
  method outperforms not only the training-free counterparts, but also those
  fine-tuned with millions of additional data samples, and sets new
  state-of-the-art records for both zero-shot semantic and referring image
  segmentation tasks. Specifically, we improve the current record by 28.8, 16.0,
  and 6.9 mIoU on Pascal VOC, COCO Object, and Pascal Context.
paper_authors._id: 657a721feaef97ff4c12889c
paper_authors.name: Shuyang Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-14T14:16:48.279Z
paper_authors._id: 657b0b4c7d962a720ec95b69
paper_authors.avatarUrl: /avatars/9d8dec39d07410a8697828b1dde15366.svg
paper_authors.fullname: Shuyang Sun
paper_authors.type: user
paper_authors.user: kevinssy
paper_authors._id: 657a721feaef97ff4c12889d
paper_authors.name: Runjia Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-14T13:43:07.007Z
paper_authors._id: 638e29cf319f9c746b87ad4b
paper_authors.avatarUrl: /avatars/70cac8d47847c389eb0393051a64c4a4.svg
paper_authors.fullname: Runjia Li
paper_authors.type: user
paper_authors.user: liguang0115
paper_authors._id: 657a721feaef97ff4c12889e
paper_authors.name: Philip Torr
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-14T13:43:13.235Z
paper_authors._id: 6565ed28a5ec0231cb07225f
paper_authors.avatarUrl: /avatars/7f95bba9aa7811d56eecb380827abfac.svg
paper_authors.fullname: prof philip torr
paper_authors.type: user
paper_authors.user: philiptorr
paper_authors._id: 657a721feaef97ff4c12889f
paper_authors.name: Xiuye Gu
paper_authors._id: 657a721feaef97ff4c1288a0
paper_authors.name: Siyang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-18T08:48:26.579Z
paper_authors._id: 657cf8e219ca6a5e92455183
paper_authors.avatarUrl: /avatars/d4feb13fb9415204ff2a79bfeffb4cb0.svg
paper_authors.fullname: Siyang Li
paper_authors.type: user
paper_authors.user: siyangli
paper_title: CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor
paper_summary: Existing open-vocabulary image segmentation methods require a fine-tuning
  step on mask annotations and/or image-text datasets. Mask labels are
  labor-intensive, which limits the number of categories in segmentation
  datasets. As a result, the open-vocabulary capacity of pre-trained VLMs is
  severely reduced after fine-tuning. However, without fine-tuning, VLMs trained
  under weak image-text supervision tend to make suboptimal mask predictions when
  there are text queries referring to non-existing concepts in the image. To
  alleviate these issues, we introduce a novel recurrent framework that
  progressively filters out irrelevant texts and enhances mask quality without
  training efforts. The recurrent unit is a two-stage segmenter built upon a VLM
  with frozen weights. Thus, our model retains the VLM's broad vocabulary space
  and strengthens its segmentation capability. Experimental results show that our
  method outperforms not only the training-free counterparts, but also those
  fine-tuned with millions of additional data samples, and sets new
  state-of-the-art records for both zero-shot semantic and referring image
  segmentation tasks. Specifically, we improve the current record by 28.8, 16.0,
  and 6.9 mIoU on Pascal VOC, COCO Object, and Pascal Context.
paper_ai_summary: A recurrent framework improves open-vocabulary image segmentation by filtering irrelevant texts and enhancing mask quality without fine-tuning.
paper_ai_keywords: open-vocabulary image segmentation fine-tuning mask annotations image-text datasets mask predictions text queries recurrent framework two-stage segmenter VLM frozen weights zero-shot semantic referring image segmentation Pascal VOC COCO Object Pascal Context mIoU