Semantic-SAM: Segment and Recognize Anything at Any Granularity
In this paper, we introduce Semantic-SAM, a universal image segmentation
model to enable segment and recognize anything at any desired granularity. Our
model offers two key advantages: semantic-awareness and granularity-abundance.
To achieve semantic-awareness, we consolidate multiple datasets across three
granularities and introduce decoupled classification for objects and parts.
This allows our model to capture rich semantic information. For the
multi-granularity capability, we propose a multi-choice learning scheme during
training, enabling each click to generate masks at multiple levels that
correspond to multiple ground-truth masks. Notably, this work represents the
first attempt to jointly train a model on SA-1B, generic, and part segmentation
datasets. Experimental results and visualizations demonstrate that our model
successfully achieves semantic-awareness and granularity-abundance.
Furthermore, combining SA-1B training with other segmentation tasks, such as
panoptic and part segmentation, leads to performance improvements. We will
provide code and a demo for further exploration and evaluation.
64acc4be5b181f6cca9bb732
Feng Li
64acc4be5b181f6cca9bb733
Hao Zhang
claimed_verified
2023-07-12T07:29:24.713Z
642e5a066748dd4f8eebcd16
https://cdn-avatars.huggingface.co/v1/production/uploads/642e5a066748dd4f8eebcd16/7ZYI0_8Z8cfsiG6r7_Y-p.jpeg
Hao Zhang
user
Haozhangcx
64acc4be5b181f6cca9bb734
Peize Sun
claimed_verified
2023-07-11T07:53:53.159Z
640dc9bf8512ec51d7f0ac1a
https://cdn-avatars.huggingface.co/v1/production/uploads/640dc9bf8512ec51d7f0ac1a/sT4rdEoQbzfW6D3xDVdqt.jpeg
peizesun
user
peizesun
64acc4be5b181f6cca9bb735
Xueyan Zou
admin_assigned
2023-07-11T09:57:25.506Z
62520988818a5dc29ab91d6f
https://cdn-avatars.huggingface.co/v1/production/uploads/1671670066375-62520988818a5dc29ab91d6f.png
Xueyan Zou
user
xueyanz
64acc4be5b181f6cca9bb736
Shilong Liu
admin_assigned
2023-07-11T09:57:56.115Z
638b13c0c1d591879698f4e2
https://cdn-avatars.huggingface.co/v1/production/uploads/638b13c0c1d591879698f4e2/X8X4EWMXuzhBpG62wO2xS.jpeg
Shilong Liu
user
ShilongLiu
64acc4be5b181f6cca9bb737
Jianwei Yang
admin_assigned
2023-07-11T09:58:40.694Z
6125df7f25027fb1ea9c7a41
https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
Jianwei Yang
user
jw2yang
64acc4be5b181f6cca9bb738
Chunyuan Li
claimed_verified
2023-07-11T07:53:48.140Z
62aba526cae4462c0c6caa0f
/avatars/430560ec2c2547f819225769ab432f30.svg
Chunyuan Li
user
Chunyuan24
64acc4be5b181f6cca9bb739
Lei Zhang
64acc4be5b181f6cca9bb73a
Jianfeng Gao
admin_assigned
2023-07-11T10:01:09.961Z
641904caf9d6f1d772ec7af7
/avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
Jianfeng Gao
user
wyngjf
Semantic-SAM: Segment and Recognize Anything at Any Granularity
In this paper, we introduce Semantic-SAM, a universal image segmentation
model to enable segment and recognize anything at any desired granularity. Our
model offers two key advantages: semantic-awareness and granularity-abundance.
To achieve semantic-awareness, we consolidate multiple datasets across three
granularities and introduce decoupled classification for objects and parts.
This allows our model to capture rich semantic information. For the
multi-granularity capability, we propose a multi-choice learning scheme during
training, enabling each click to generate masks at multiple levels that
correspond to multiple ground-truth masks. Notably, this work represents the
first attempt to jointly train a model on SA-1B, generic, and part segmentation
datasets. Experimental results and visualizations demonstrate that our model
successfully achieves semantic-awareness and granularity-abundance.
Furthermore, combining SA-1B training with other segmentation tasks, such as
panoptic and part segmentation, leads to performance improvements. We will
provide code and a demo for further exploration and evaluation.
Semantic-SAM, a universal image segmentation model, achieves semantic-awareness and multi-granularity by introducing decoupled classification and multi-choice learning, improving performance across various segmentation tasks.
semantic-awareness granularity-abundance decoupled classification multi-choice learning SA-1B datasets panoptic segmentation part segmentation