SAM-CP: Marrying SAM with Composable Prompts for Versatile Segmentation
Pengfei Chen Lingxi Xie Xinyue Huo Xuehui Yu XIAOPENG ZHANG Yingfei Sun Zhenjun Han Qi Tian
The Segment Anything model (SAM) has shown a generalized ability to group image pixels into patches, but applying it to semantic-aware segmentation still faces major challenges. This paper presents SAM-CP, a simple approach that establishes two types of composable prompts beyond SAM and composes them for versatile segmentation. Specifically, given a set of classes (in texts) and a set of SAM patches, the Type-I prompt judges whether a SAM patch aligns with a text label, and the Type-II prompt judges whether two SAM patches with the same text label also belong to the same instance. To decrease the complexity in dealing with a large number of semantic classes and patches, we establish a unified framework that calculates the affinity between (semantic and instance) queries and SAM patches, and then merges patches with high affinity to the query. Experiments show that SAM-CP achieves semantic, instance, and panoptic segmentation in both open and closed domains. In particular, it achieves state-of-the-art performance in open-vocabulary segmentation. Our research offers a novel and generalized methodology for equipping vision foundation models like SAM with multi-grained semantic perception abilities. Codes are released on https://github.com/ucas-vg/SAM-CP.
Versatile Segmentation, Composable Prompts, Open-vocabulary
We design a new paradigm of vision perception: two composable prompts based on the vision foundation model SAM to achieve versatile segmentation tasks in both open and closed domains.