title: Semantic-SAM: Segment and Recognize Anything at Any Granularity
summary: In this paper, we introduce Semantic-SAM, a universal image segmentation
  model to enable segment and recognize anything at any desired granularity. Our
  model offers two key advantages: semantic-awareness and granularity-abundance.
  To achieve semantic-awareness, we consolidate multiple datasets across three
  granularities and introduce decoupled classification for objects and parts.
  This allows our model to capture rich semantic information. For the
  multi-granularity capability, we propose a multi-choice learning scheme during
  training, enabling each click to generate masks at multiple levels that
  correspond to multiple ground-truth masks. Notably, this work represents the
  first attempt to jointly train a model on SA-1B, generic, and part segmentation
  datasets. Experimental results and visualizations demonstrate that our model
  successfully achieves semantic-awareness and granularity-abundance.
  Furthermore, combining SA-1B training with other segmentation tasks, such as
  panoptic and part segmentation, leads to performance improvements. We will
  provide code and a demo for further exploration and evaluation.
paper_authors._id: 64acc4be5b181f6cca9bb732
paper_authors.name: Feng Li
paper_authors._id: 64acc4be5b181f6cca9bb733
paper_authors.name: Hao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-12T07:29:24.713Z
paper_authors._id: 642e5a066748dd4f8eebcd16
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642e5a066748dd4f8eebcd16/7ZYI0_8Z8cfsiG6r7_Y-p.jpeg
paper_authors.fullname: Hao Zhang
paper_authors.type: user
paper_authors.user: Haozhangcx
paper_authors._id: 64acc4be5b181f6cca9bb734
paper_authors.name: Peize Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-11T07:53:53.159Z
paper_authors._id: 640dc9bf8512ec51d7f0ac1a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/640dc9bf8512ec51d7f0ac1a/sT4rdEoQbzfW6D3xDVdqt.jpeg
paper_authors.fullname: peizesun
paper_authors.type: user
paper_authors.user: peizesun
paper_authors._id: 64acc4be5b181f6cca9bb735
paper_authors.name: Xueyan Zou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-11T09:57:25.506Z
paper_authors._id: 62520988818a5dc29ab91d6f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1671670066375-62520988818a5dc29ab91d6f.png
paper_authors.fullname: Xueyan Zou
paper_authors.type: user
paper_authors.user: xueyanz
paper_authors._id: 64acc4be5b181f6cca9bb736
paper_authors.name: Shilong Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-11T09:57:56.115Z
paper_authors._id: 638b13c0c1d591879698f4e2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638b13c0c1d591879698f4e2/X8X4EWMXuzhBpG62wO2xS.jpeg
paper_authors.fullname: Shilong Liu
paper_authors.type: user
paper_authors.user: ShilongLiu
paper_authors._id: 64acc4be5b181f6cca9bb737
paper_authors.name: Jianwei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-11T09:58:40.694Z
paper_authors._id: 6125df7f25027fb1ea9c7a41
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
paper_authors.fullname: Jianwei Yang
paper_authors.type: user
paper_authors.user: jw2yang
paper_authors._id: 64acc4be5b181f6cca9bb738
paper_authors.name: Chunyuan Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-11T07:53:48.140Z
paper_authors._id: 62aba526cae4462c0c6caa0f
paper_authors.avatarUrl: /avatars/430560ec2c2547f819225769ab432f30.svg
paper_authors.fullname: Chunyuan Li
paper_authors.type: user
paper_authors.user: Chunyuan24
paper_authors._id: 64acc4be5b181f6cca9bb739
paper_authors.name: Lei Zhang
paper_authors._id: 64acc4be5b181f6cca9bb73a
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-11T10:01:09.961Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_title: Semantic-SAM: Segment and Recognize Anything at Any Granularity
paper_summary: In this paper, we introduce Semantic-SAM, a universal image segmentation
  model to enable segment and recognize anything at any desired granularity. Our
  model offers two key advantages: semantic-awareness and granularity-abundance.
  To achieve semantic-awareness, we consolidate multiple datasets across three
  granularities and introduce decoupled classification for objects and parts.
  This allows our model to capture rich semantic information. For the
  multi-granularity capability, we propose a multi-choice learning scheme during
  training, enabling each click to generate masks at multiple levels that
  correspond to multiple ground-truth masks. Notably, this work represents the
  first attempt to jointly train a model on SA-1B, generic, and part segmentation
  datasets. Experimental results and visualizations demonstrate that our model
  successfully achieves semantic-awareness and granularity-abundance.
  Furthermore, combining SA-1B training with other segmentation tasks, such as
  panoptic and part segmentation, leads to performance improvements. We will
  provide code and a demo for further exploration and evaluation.
paper_ai_keywords: Semantic-SAM image segmentation decoupled classification semantic-awareness parallel-awareness granularity-abundance multi-granularity capability multi-choice learning scheme confidence intervals panoptic segmentation part segmentation