SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial
  Understanding
The landscape of publicly available vision foundation models (VFMs), such as
CLIP and Segment Anything Model (SAM), is expanding rapidly. VFMs are endowed
with distinct capabilities stemming from their pre-training objectives. For
instance, CLIP excels in semantic understanding, while SAM specializes in
spatial understanding for segmentation. In this work, we introduce a simple
recipe to efficiently merge VFMs into a unified model that assimilates their
expertise. Our proposed method integrates multi-task learning, continual
learning techniques, and teacher-student distillation. This strategy entails
significantly less computational cost compared to traditional multi-task
training from scratch. Additionally, it only demands a small fraction of the
pre-training datasets that were initially used to train individual models. By
applying our method to SAM and CLIP, we derive SAM-CLIP: a unified model that
amalgamates the strengths of SAM and CLIP into a single backbone, making it apt
for edge device applications. We show that SAM-CLIP learns richer visual
representations, equipped with both localization and semantic features,
suitable for a broad range of vision tasks. SAM-CLIP obtains improved
performance on several head probing tasks when compared with SAM and CLIP. We
further show that SAM-CLIP not only retains the foundational strengths of its
precursor models but also introduces synergistic functionalities, most notably
in zero-shot semantic segmentation, where SAM-CLIP establishes new
state-of-the-art results on 5 benchmarks. It outperforms previous models that
are specifically designed for this task by a large margin, including +6.8% and
+5.9% mean IoU improvement on Pascal-VOC and COCO-Stuff datasets, respectively.
6538a1f91bdb3c40db9969e6
Haoxiang Wang
admin_assigned
2023-10-26T11:37:33.333Z
638fb8cf2380ffd99caf8c2a
https://cdn-avatars.huggingface.co/v1/production/uploads/638fb8cf2380ffd99caf8c2a/7juUomHJ4gH1HIJj43qE6.jpeg
Haoxiang Wang
user
Haoxiang-Wang
6538a1f91bdb3c40db9969e7
Pavan Kumar Anasosalu Vasu
claimed_verified
2024-07-26T07:41:34.534Z
65e000cce1823ce0df17a3d4
/avatars/b50c2d2c1f3ad1114867e199fd578a29.svg
Pavan Kumar Vasu
user
pavankumarvasu
6538a1f91bdb3c40db9969e8
Fartash Faghri
claimed_verified
2024-04-26T15:29:13.359Z
65f8945a3ba58880f9794a11
/avatars/fde0ea1c20f89b3a83dbefb24e728d61.svg
Fartash Faghri
user
fartashf
6538a1f91bdb3c40db9969e9
Raviteja Vemulapalli
6538a1f91bdb3c40db9969ea
Mehrdad Farajtabar
admin_assigned
2023-10-26T11:38:25.644Z
62d188a6d50433c35eb14020
/avatars/fc3d149a27bc8396010a2a02148e8ca0.svg
Mehrdad Farajtabar
user
mfarajtabar
6538a1f91bdb3c40db9969eb
Sachin Mehta
admin_assigned
2023-10-26T11:38:32.440Z
64f792652003abb61bf5c1b8
/avatars/9721810de3519ece50b059c154baa9ca.svg
Sachin Mehta
user
sacmehta
6538a1f91bdb3c40db9969ec
Mohammad Rastegari
6538a1f91bdb3c40db9969ed
Oncel Tuzel
6538a1f91bdb3c40db9969ee
Hadi Pouransari
claimed_verified
2024-02-19T09:35:47.481Z
653a8e65c75dc136bfb5b0f8
/avatars/ef252794236ce0fe2debf12773c95bb2.svg
Hadi Pouransari
user
hpouransari
SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial
  Understanding
The landscape of publicly available vision foundation models (VFMs), such as
CLIP and Segment Anything Model (SAM), is expanding rapidly. VFMs are endowed
with distinct capabilities stemming from their pre-training objectives. For
instance, CLIP excels in semantic understanding, while SAM specializes in
spatial understanding for segmentation. In this work, we introduce a simple
recipe to efficiently merge VFMs into a unified model that assimilates their
expertise. Our proposed method integrates multi-task learning, continual
learning techniques, and teacher-student distillation. This strategy entails
significantly less computational cost compared to traditional multi-task
training from scratch. Additionally, it only demands a small fraction of the
pre-training datasets that were initially used to train individual models. By
applying our method to SAM and CLIP, we derive SAM-CLIP: a unified model that
amalgamates the strengths of SAM and CLIP into a single backbone, making it apt
for edge device applications. We show that SAM-CLIP learns richer visual
representations, equipped with both localization and semantic features,
suitable for a broad range of vision tasks. SAM-CLIP obtains improved
performance on several head probing tasks when compared with SAM and CLIP. We
further show that SAM-CLIP not only retains the foundational strengths of its
precursor models but also introduces synergistic functionalities, most notably
in zero-shot semantic segmentation, where SAM-CLIP establishes new
state-of-the-art results on 5 benchmarks. It outperforms previous models that
are specifically designed for this task by a large margin, including +6.8% and
+5.9% mean IoU improvement on Pascal-VOC and COCO-Stuff datasets, respectively.
A unified model combining CLIP and SAM using multi-task learning, continual learning, and distillation techniques achieves state-of-the-art zero-shot semantic segmentation performance with reduced computational and data requirements.
vision foundation models (VFMs) CLIP Segment Anything Model (SAM) multi-task learning continual learning teacher-student distillation visual representations localization semantic features zero-shot semantic segmentation mean IoU Pascal-VOC COCO-Stuff