title: Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes
    Interactively
summary: The CLIP and Segment Anything Model (SAM) are remarkable vision foundation
  models (VFMs). SAM excels in segmentation tasks across diverse domains, while
  CLIP is renowned for its zero-shot recognition capabilities. This paper
  presents an in-depth exploration of integrating these two models into a unified
  framework. Specifically, we introduce the Open-Vocabulary SAM, a SAM-inspired
  model designed for simultaneous interactive segmentation and recognition,
  leveraging two unique knowledge transfer modules: SAM2CLIP and CLIP2SAM. The
  former adapts SAM's knowledge into the CLIP via distillation and learnable
  transformer adapters, while the latter transfers CLIP knowledge into SAM,
  enhancing its recognition capabilities. Extensive experiments on various
  datasets and detectors show the effectiveness of Open-Vocabulary SAM in both
  segmentation and recognition tasks, significantly outperforming the naive
  baselines of simply combining SAM and CLIP. Furthermore, aided with image
  classification data training, our method can segment and recognize
  approximately 22,000 classes.
paper_authors._id: 659b62626da3461e2849234a
paper_authors.name: Haobo Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-08T11:45:19.748Z
paper_authors._id: 6391e41f2e73987364e6bcb2
paper_authors.avatarUrl: /avatars/d09a9ee329bb8c3a9e2929d67d24e97d.svg
paper_authors.fullname: Haobo Yuan
paper_authors.type: user
paper_authors.user: HarborYuan
paper_authors._id: 659b62626da3461e2849234b
paper_authors.name: Xiangtai Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-08T11:45:43.664Z
paper_authors._id: 63958b4414513eaf9029ebf1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U1g5H071pWRswGAG9UTpo.png
paper_authors.fullname: Xiangtai Li
paper_authors.type: user
paper_authors.user: LXT
paper_authors._id: 659b62626da3461e2849234c
paper_authors.name: Chong Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-08T08:47:30.592Z
paper_authors._id: 642bfdbbc885078517154abb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642bfdbbc885078517154abb/lLmc8_3VnTyk4qOvl-omI.jpeg
paper_authors.fullname: Chong Zhou
paper_authors.type: user
paper_authors.user: chongzhou
paper_authors._id: 659b62626da3461e2849234d
paper_authors.name: Yining Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-08T11:46:10.233Z
paper_authors._id: 649988726677f66c2b486392
paper_authors.avatarUrl: /avatars/b649a77370660e129726e29504daba34.svg
paper_authors.fullname: Yining Li
paper_authors.type: user
paper_authors.user: ly015
paper_authors._id: 659b62626da3461e2849234e
paper_authors.name: Kai Chen
paper_authors._id: 659b62626da3461e2849234f
paper_authors.name: Chen Change Loy
paper_title: Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes
    Interactively
paper_summary: The CLIP and Segment Anything Model (SAM) are remarkable vision foundation
  models (VFMs). SAM excels in segmentation tasks across diverse domains, while
  CLIP is renowned for its zero-shot recognition capabilities. This paper
  presents an in-depth exploration of integrating these two models into a unified
  framework. Specifically, we introduce the Open-Vocabulary SAM, a SAM-inspired
  model designed for simultaneous interactive segmentation and recognition,
  leveraging two unique knowledge transfer modules: SAM2CLIP and CLIP2SAM. The
  former adapts SAM's knowledge into the CLIP via distillation and learnable
  transformer adapters, while the latter transfers CLIP knowledge into SAM,
  enhancing its recognition capabilities. Extensive experiments on various
  datasets and detectors show the effectiveness of Open-Vocabulary SAM in both
  segmentation and recognition tasks, significantly outperforming the naive
  baselines of simply combining SAM and CLIP. Furthermore, aided with image
  classification data training, our method can segment and recognize
  approximately 22,000 classes.
paper_ai_summary: A unified framework combining SAM and CLIP through knowledge transfer improves performance in both segmentation and recognition tasks.
paper_ai_keywords: CLIP Segment Anything Model (SAM) vision foundation models (VFMs) interactive segmentation open-vocabulary SAM SAM2CLIP CLIP2SAM knowledge transfer transformer adapters image classification