CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor
Existing open-vocabulary image segmentation methods require a fine-tuning
step on mask annotations and/or image-text datasets. Mask labels are
labor-intensive, which limits the number of categories in segmentation
datasets. As a result, the open-vocabulary capacity of pre-trained VLMs is
severely reduced after fine-tuning. However, without fine-tuning, VLMs trained
under weak image-text supervision tend to make suboptimal mask predictions when
there are text queries referring to non-existing concepts in the image. To
alleviate these issues, we introduce a novel recurrent framework that
progressively filters out irrelevant texts and enhances mask quality without
training efforts. The recurrent unit is a two-stage segmenter built upon a VLM
with frozen weights. Thus, our model retains the VLM's broad vocabulary space
and strengthens its segmentation capability. Experimental results show that our
method outperforms not only the training-free counterparts, but also those
fine-tuned with millions of additional data samples, and sets new
state-of-the-art records for both zero-shot semantic and referring image
segmentation tasks. Specifically, we improve the current record by 28.8, 16.0,
and 6.9 mIoU on Pascal VOC, COCO Object, and Pascal Context.
657a721feaef97ff4c12889c
Shuyang Sun
claimed_verified
2023-12-14T14:16:48.279Z
657b0b4c7d962a720ec95b69
/avatars/9d8dec39d07410a8697828b1dde15366.svg
Shuyang Sun
user
kevinssy
657a721feaef97ff4c12889d
Runjia Li
admin_assigned
2023-12-14T13:43:07.007Z
638e29cf319f9c746b87ad4b
/avatars/70cac8d47847c389eb0393051a64c4a4.svg
Runjia Li
user
liguang0115
657a721feaef97ff4c12889e
Philip Torr
admin_assigned
2023-12-14T13:43:13.235Z
6565ed28a5ec0231cb07225f
/avatars/7f95bba9aa7811d56eecb380827abfac.svg
prof philip torr
user
philiptorr
657a721feaef97ff4c12889f
Xiuye Gu
657a721feaef97ff4c1288a0
Siyang Li
claimed_verified
2023-12-18T08:48:26.579Z
657cf8e219ca6a5e92455183
/avatars/d4feb13fb9415204ff2a79bfeffb4cb0.svg
Siyang Li
user
siyangli
CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor
Existing open-vocabulary image segmentation methods require a fine-tuning
step on mask annotations and/or image-text datasets. Mask labels are
labor-intensive, which limits the number of categories in segmentation
datasets. As a result, the open-vocabulary capacity of pre-trained VLMs is
severely reduced after fine-tuning. However, without fine-tuning, VLMs trained
under weak image-text supervision tend to make suboptimal mask predictions when
there are text queries referring to non-existing concepts in the image. To
alleviate these issues, we introduce a novel recurrent framework that
progressively filters out irrelevant texts and enhances mask quality without
training efforts. The recurrent unit is a two-stage segmenter built upon a VLM
with frozen weights. Thus, our model retains the VLM's broad vocabulary space
and strengthens its segmentation capability. Experimental results show that our
method outperforms not only the training-free counterparts, but also those
fine-tuned with millions of additional data samples, and sets new
state-of-the-art records for both zero-shot semantic and referring image
segmentation tasks. Specifically, we improve the current record by 28.8, 16.0,
and 6.9 mIoU on Pascal VOC, COCO Object, and Pascal Context.
A recurrent framework improves open-vocabulary image segmentation by filtering irrelevant texts and enhancing mask quality without fine-tuning.
open-vocabulary image segmentation fine-tuning mask annotations image-text datasets mask predictions text queries recurrent framework two-stage segmenter VLM frozen weights zero-shot semantic referring image segmentation Pascal VOC COCO Object Pascal Context mIoU