title: Segment Anything in High Quality
summary: The recent Segment Anything Model (SAM) represents a big leap in scaling up
  segmentation models, allowing for powerful zero-shot capabilities and flexible
  prompting. Despite being trained with 1.1 billion masks, SAM's mask prediction
  quality falls short in many cases, particularly when dealing with objects that
  have intricate structures. We propose HQ-SAM, equipping SAM with the ability to
  accurately segment any object, while maintaining SAM's original promptable
  design, efficiency, and zero-shot generalizability. Our careful design reuses
  and preserves the pre-trained model weights of SAM, while only introducing
  minimal additional parameters and computation. We design a learnable
  High-Quality Output Token, which is injected into SAM's mask decoder and is
  responsible for predicting the high-quality mask. Instead of only applying it
  on mask-decoder features, we first fuse them with early and final ViT features
  for improved mask details. To train our introduced learnable parameters, we
  compose a dataset of 44K fine-grained masks from several sources. HQ-SAM is
  only trained on the introduced detaset of 44k masks, which takes only 4 hours
  on 8 GPUs. We show the efficacy of HQ-SAM in a suite of 9 diverse segmentation
  datasets across different downstream tasks, where 7 out of them are evaluated
  in a zero-shot transfer protocol. Our code and models will be released at
  https://github.com/SysCV/SAM-HQ.
paper_authors._id: 647d5a3f1f878439e203503c
paper_authors.name: Lei Ke
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-05T15:20:25.078Z
paper_authors._id: 6258a6455ea3a0a9b6de3f22
paper_authors.avatarUrl: /avatars/6eeed72a97fb24465e5e65583fbe50cf.svg
paper_authors.fullname: Lei Ke
paper_authors.type: user
paper_authors.user: lkeab
paper_authors._id: 647d5a3f1f878439e203503d
paper_authors.name: Mingqiao Ye
paper_authors._id: 647d5a3f1f878439e203503e
paper_authors.name: Martin Danelljan
paper_authors._id: 647d5a3f1f878439e203503f
paper_authors.name: Yifan Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-05T15:20:30.370Z
paper_authors._id: 647dd7341a1fcad2fdc0bf9b
paper_authors.avatarUrl: /avatars/5a02c8c51e1e1dd1079a6c68fc54b5e7.svg
paper_authors.fullname: Yifan Liu
paper_authors.type: user
paper_authors.user: YifanLiu
paper_authors._id: 647d5a3f1f878439e2035040
paper_authors.name: Yu-Wing Tai
paper_authors._id: 647d5a3f1f878439e2035041
paper_authors.name: Chi-Keung Tang
paper_authors._id: 647d5a3f1f878439e2035042
paper_authors.name: Fisher Yu
paper_title: Segment Anything in High Quality
paper_summary: The recent Segment Anything Model (SAM) represents a big leap in scaling up
  segmentation models, allowing for powerful zero-shot capabilities and flexible
  prompting. Despite being trained with 1.1 billion masks, SAM's mask prediction
  quality falls short in many cases, particularly when dealing with objects that
  have intricate structures. We propose HQ-SAM, equipping SAM with the ability to
  accurately segment any object, while maintaining SAM's original promptable
  design, efficiency, and zero-shot generalizability. Our careful design reuses
  and preserves the pre-trained model weights of SAM, while only introducing
  minimal additional parameters and computation. We design a learnable
  High-Quality Output Token, which is injected into SAM's mask decoder and is
  responsible for predicting the high-quality mask. Instead of only applying it
  on mask-decoder features, we first fuse them with early and final ViT features
  for improved mask details. To train our introduced learnable parameters, we
  compose a dataset of 44K fine-grained masks from several sources. HQ-SAM is
  only trained on the introduced detaset of 44k masks, which takes only 4 hours
  on 8 GPUs. We show the efficacy of HQ-SAM in a suite of 9 diverse segmentation
  datasets across different downstream tasks, where 7 out of them are evaluated
  in a zero-shot transfer protocol. Our code and models will be released at
  https://github.com/SysCV/SAM-HQ.
paper_ai_summary: HQ-SAM enhances the segmentation capabilities of SAM by introducing a learnable output token that leverages additional Vit features, improving mask quality with minimal computational overhead.
paper_ai_keywords: Segment Anything Model SAM zero-shot capabilities promptable design mask prediction mask decoder learnable High-Quality Output Token ViT features fine-grained masks zero-shot transfer