SAM-CP: Marrying SAM with Composable Prompts for Versatile Segmentation
<|reference_start|>SAM-CP: Marrying SAM with Composable Prompts for Versatile Segmentation: The Segment Anything model (SAM) has shown a generalized ability to group image pixels into patches, but applying it to semantic-aware segmentation still faces major challenges. This paper presents SAM-CP, a simple approach that establishes two types of composable prompts beyond SAM and composes them for versatile segmentation. Specifically, given a set of classes (in texts) and a set of SAM patches, the Type-I prompt judges whether a SAM patch aligns with a text label, and the Type-II prompt judges whether two SAM patches with the same text label also belong to the same instance. To decrease the complexity in dealing with a large number of semantic classes and patches, we establish a unified framework that calculates the affinity between (semantic and instance) queries and SAM patches and merges patches with high affinity to the query. Experiments show that SAM-CP achieves semantic, instance, and panoptic segmentation in both open and closed domains. In particular, it achieves state-of-the-art performance in open-vocabulary segmentation. Our research offers a novel and generalized methodology for equipping vision foundation models like SAM with multi-grained semantic perception abilities.<|reference_end|>
@article{chen2024sam-cp:,
    title={SAM-CP: Marrying SAM with Composable Prompts for Versatile Segmentation},
    author={Pengfei Chen, Lingxi Xie, Xinyue Huo, Xuehui Yu, Xiaopeng Zhang,
  Yingfei Sun, Zhenjun Han, Qi Tian},
    journal={arXiv preprint arXiv:2407.16682},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.16682},
    primaryClass={cs.CV}
}