ITACLIP: Boosting Training-Free Semantic Segmentation with Image, Text,
  and Architectural Enhancements
Recent advances in foundational Vision Language Models (VLMs) have reshaped
the evaluation paradigm in computer vision tasks. These foundational models,
especially CLIP, have accelerated research in open-vocabulary computer vision
tasks, including Open-Vocabulary Semantic Segmentation (OVSS). Although the
initial results are promising, the dense prediction capabilities of VLMs still
require further improvement. In this study, we enhance the semantic
segmentation performance of CLIP by introducing new modules and modifications:
1) architectural changes in the last layer of ViT and the incorporation of
attention maps from the middle layers with the last layer, 2) Image
Engineering: applying data augmentations to enrich input image representations,
and 3) using Large Language Models (LLMs) to generate definitions and synonyms
for each class name to leverage CLIP's open-vocabulary capabilities. Our
training-free method, ITACLIP, outperforms current state-of-the-art approaches
on segmentation benchmarks such as COCO-Stuff, COCO-Object, Pascal Context, and
Pascal VOC. Our code is available at https://github.com/m-arda-aydn/ITACLIP.
673d97e8f48649bd99d56715
M. Arda Aydın
claimed_verified
2024-11-20T10:08:20.077Z
63923c2a5ece53ea0b364c8a
/avatars/d856933ecd8966ef72f1e67c6ec166b6.svg
Mustafa Arda Aydın
user
aydnarda
673d97e8f48649bd99d56716
Efe Mert Çırpar
admin_assigned
2024-11-20T13:33:17.654Z
65f411590718933804abbf56
/avatars/eb562172c0587c89288cfc5f32205a41.svg
Efe Mert Çırpar
user
efemcirpar
673d97e8f48649bd99d56717
Elvin Abdinli
claimed_verified
2024-11-21T08:09:47.221Z
673e20b9635a89280bfd901a
/avatars/95cb42c7bc30a9c23eadd51bf961454d.svg
Elvin Abdinli
user
elvinabdinli
673d97e8f48649bd99d56718
Gozde Unal
admin_assigned
2024-11-20T13:33:26.419Z
644b88b00fbe4830f192583f
/avatars/0e1cbab70d9b3ad4340985ee3087a8a7.svg
Gozde Unal
user
gozy
673d97e8f48649bd99d56719
Yusuf H. Sahin
admin_assigned
2024-11-20T13:33:31.767Z
6454104b9dcddd329ae73fc6
/avatars/996ee479de8efa7c1deafe8de634e94b.svg
Yusuf H Sahin
user
sahinyu
ITACLIP: Boosting Training-Free Semantic Segmentation with Image, Text,
  and Architectural Enhancements
Recent advances in foundational Vision Language Models (VLMs) have reshaped
the evaluation paradigm in computer vision tasks. These foundational models,
especially CLIP, have accelerated research in open-vocabulary computer vision
tasks, including Open-Vocabulary Semantic Segmentation (OVSS). Although the
initial results are promising, the dense prediction capabilities of VLMs still
require further improvement. In this study, we enhance the semantic
segmentation performance of CLIP by introducing new modules and modifications:
1) architectural changes in the last layer of ViT and the incorporation of
attention maps from the middle layers with the last layer, 2) Image
Engineering: applying data augmentations to enrich input image representations,
and 3) using Large Language Models (LLMs) to generate definitions and synonyms
for each class name to leverage CLIP's open-vocabulary capabilities. Our
training-free method, ITACLIP, outperforms current state-of-the-art approaches
on segmentation benchmarks such as COCO-Stuff, COCO-Object, Pascal Context, and
Pascal VOC. Our code is available at https://github.com/m-arda-aydn/ITACLIP.
ITACLIP enhances CLIP's semantic segmentation through architectural modifications, data augmentation, and language model-generated definitions, outperforming existing benchmarks.
Vision Language Models CLIP Open-Vocabulary Semantic Segmentation ViT attention maps Large Language Models ITACLIP COCO-Stuff COCO-Object Pascal Context Pascal VOC