title: SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial
    Understanding
summary: The landscape of publicly available vision foundation models (VFMs), such as
  CLIP and Segment Anything Model (SAM), is expanding rapidly. VFMs are endowed
  with distinct capabilities stemming from their pre-training objectives. For
  instance, CLIP excels in semantic understanding, while SAM specializes in
  spatial understanding for segmentation. In this work, we introduce a simple
  recipe to efficiently merge VFMs into a unified model that assimilates their
  expertise. Our proposed method integrates multi-task learning, continual
  learning techniques, and teacher-student distillation. This strategy entails
  significantly less computational cost compared to traditional multi-task
  training from scratch. Additionally, it only demands a small fraction of the
  pre-training datasets that were initially used to train individual models. By
  applying our method to SAM and CLIP, we derive SAM-CLIP: a unified model that
  amalgamates the strengths of SAM and CLIP into a single backbone, making it apt
  for edge device applications. We show that SAM-CLIP learns richer visual
  representations, equipped with both localization and semantic features,
  suitable for a broad range of vision tasks. SAM-CLIP obtains improved
  performance on several head probing tasks when compared with SAM and CLIP. We
  further show that SAM-CLIP not only retains the foundational strengths of its
  precursor models but also introduces synergistic functionalities, most notably
  in zero-shot semantic segmentation, where SAM-CLIP establishes new
  state-of-the-art results on 5 benchmarks. It outperforms previous models that
  are specifically designed for this task by a large margin, including +6.8% and
  +5.9% mean IoU improvement on Pascal-VOC and COCO-Stuff datasets, respectively.
paper_authors._id: 6538a1f91bdb3c40db9969e6
paper_authors.name: Haoxiang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T11:37:33.333Z
paper_authors._id: 638fb8cf2380ffd99caf8c2a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638fb8cf2380ffd99caf8c2a/7juUomHJ4gH1HIJj43qE6.jpeg
paper_authors.fullname: Haoxiang Wang
paper_authors.type: user
paper_authors.user: Haoxiang-Wang
paper_authors._id: 6538a1f91bdb3c40db9969e7
paper_authors.name: Pavan Kumar Anasosalu Vasu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-26T07:41:34.534Z
paper_authors._id: 65e000cce1823ce0df17a3d4
paper_authors.avatarUrl: /avatars/b50c2d2c1f3ad1114867e199fd578a29.svg
paper_authors.fullname: Pavan Kumar Vasu
paper_authors.type: user
paper_authors.user: pavankumarvasu
paper_authors._id: 6538a1f91bdb3c40db9969e8
paper_authors.name: Fartash Faghri
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-26T15:29:13.359Z
paper_authors._id: 65f8945a3ba58880f9794a11
paper_authors.avatarUrl: /avatars/fde0ea1c20f89b3a83dbefb24e728d61.svg
paper_authors.fullname: Fartash Faghri
paper_authors.type: user
paper_authors.user: fartashf
paper_authors._id: 6538a1f91bdb3c40db9969e9
paper_authors.name: Raviteja Vemulapalli
paper_authors._id: 6538a1f91bdb3c40db9969ea
paper_authors.name: Mehrdad Farajtabar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T11:38:25.644Z
paper_authors._id: 62d188a6d50433c35eb14020
paper_authors.avatarUrl: /avatars/fc3d149a27bc8396010a2a02148e8ca0.svg
paper_authors.fullname: Mehrdad Farajtabar
paper_authors.type: user
paper_authors.user: mfarajtabar
paper_authors._id: 6538a1f91bdb3c40db9969eb
paper_authors.name: Sachin Mehta
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-26T11:38:32.440Z
paper_authors._id: 64f792652003abb61bf5c1b8
paper_authors.avatarUrl: /avatars/9721810de3519ece50b059c154baa9ca.svg
paper_authors.fullname: Sachin Mehta
paper_authors.type: user
paper_authors.user: sacmehta
paper_authors._id: 6538a1f91bdb3c40db9969ec
paper_authors.name: Mohammad Rastegari
paper_authors._id: 6538a1f91bdb3c40db9969ed
paper_authors.name: Oncel Tuzel
paper_authors._id: 6538a1f91bdb3c40db9969ee
paper_authors.name: Hadi Pouransari
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-19T09:35:47.481Z
paper_authors._id: 653a8e65c75dc136bfb5b0f8
paper_authors.avatarUrl: /avatars/ef252794236ce0fe2debf12773c95bb2.svg
paper_authors.fullname: Hadi Pouransari
paper_authors.type: user
paper_authors.user: hpouransari
paper_title: SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial
    Understanding
paper_summary: The landscape of publicly available vision foundation models (VFMs), such as
  CLIP and Segment Anything Model (SAM), is expanding rapidly. VFMs are endowed
  with distinct capabilities stemming from their pre-training objectives. For
  instance, CLIP excels in semantic understanding, while SAM specializes in
  spatial understanding for segmentation. In this work, we introduce a simple
  recipe to efficiently merge VFMs into a unified model that assimilates their
  expertise. Our proposed method integrates multi-task learning, continual
  learning techniques, and teacher-student distillation. This strategy entails
  significantly less computational cost compared to traditional multi-task
  training from scratch. Additionally, it only demands a small fraction of the
  pre-training datasets that were initially used to train individual models. By
  applying our method to SAM and CLIP, we derive SAM-CLIP: a unified model that
  amalgamates the strengths of SAM and CLIP into a single backbone, making it apt
  for edge device applications. We show that SAM-CLIP learns richer visual
  representations, equipped with both localization and semantic features,
  suitable for a broad range of vision tasks. SAM-CLIP obtains improved
  performance on several head probing tasks when compared with SAM and CLIP. We
  further show that SAM-CLIP not only retains the foundational strengths of its
  precursor models but also introduces synergistic functionalities, most notably
  in zero-shot semantic segmentation, where SAM-CLIP establishes new
  state-of-the-art results on 5 benchmarks. It outperforms previous models that
  are specifically designed for this task by a large margin, including +6.8% and
  +5.9% mean IoU improvement on Pascal-VOC and COCO-Stuff datasets, respectively.
paper_ai_summary: A unified model combining CLIP and SAM using multi-task learning, continual learning, and distillation techniques achieves state-of-the-art zero-shot semantic segmentation performance with reduced computational and data requirements.
paper_ai_keywords: vision foundation models (VFMs) CLIP Segment Anything Model (SAM) multi-task learning continual learning teacher-student distillation visual representations localization semantic features zero-shot semantic segmentation mean IoU Pascal-VOC COCO-Stuff