Segment Anything in High Quality
The recent Segment Anything Model (SAM) represents a big leap in scaling up
segmentation models, allowing for powerful zero-shot capabilities and flexible
prompting. Despite being trained with 1.1 billion masks, SAM's mask prediction
quality falls short in many cases, particularly when dealing with objects that
have intricate structures. We propose HQ-SAM, equipping SAM with the ability to
accurately segment any object, while maintaining SAM's original promptable
design, efficiency, and zero-shot generalizability. Our careful design reuses
and preserves the pre-trained model weights of SAM, while only introducing
minimal additional parameters and computation. We design a learnable
High-Quality Output Token, which is injected into SAM's mask decoder and is
responsible for predicting the high-quality mask. Instead of only applying it
on mask-decoder features, we first fuse them with early and final ViT features
for improved mask details. To train our introduced learnable parameters, we
compose a dataset of 44K fine-grained masks from several sources. HQ-SAM is
only trained on the introduced detaset of 44k masks, which takes only 4 hours
on 8 GPUs. We show the efficacy of HQ-SAM in a suite of 9 diverse segmentation
datasets across different downstream tasks, where 7 out of them are evaluated
in a zero-shot transfer protocol. Our code and models will be released at
https://github.com/SysCV/SAM-HQ.
647d5a3f1f878439e203503c
Lei Ke
claimed_verified
2023-06-05T15:20:25.078Z
6258a6455ea3a0a9b6de3f22
/avatars/6eeed72a97fb24465e5e65583fbe50cf.svg
Lei Ke
user
lkeab
647d5a3f1f878439e203503d
Mingqiao Ye
647d5a3f1f878439e203503e
Martin Danelljan
647d5a3f1f878439e203503f
Yifan Liu
claimed_verified
2023-06-05T15:20:30.370Z
647dd7341a1fcad2fdc0bf9b
/avatars/5a02c8c51e1e1dd1079a6c68fc54b5e7.svg
Yifan Liu
user
YifanLiu
647d5a3f1f878439e2035040
Yu-Wing Tai
647d5a3f1f878439e2035041
Chi-Keung Tang
647d5a3f1f878439e2035042
Fisher Yu
Segment Anything in High Quality
The recent Segment Anything Model (SAM) represents a big leap in scaling up
segmentation models, allowing for powerful zero-shot capabilities and flexible
prompting. Despite being trained with 1.1 billion masks, SAM's mask prediction
quality falls short in many cases, particularly when dealing with objects that
have intricate structures. We propose HQ-SAM, equipping SAM with the ability to
accurately segment any object, while maintaining SAM's original promptable
design, efficiency, and zero-shot generalizability. Our careful design reuses
and preserves the pre-trained model weights of SAM, while only introducing
minimal additional parameters and computation. We design a learnable
High-Quality Output Token, which is injected into SAM's mask decoder and is
responsible for predicting the high-quality mask. Instead of only applying it
on mask-decoder features, we first fuse them with early and final ViT features
for improved mask details. To train our introduced learnable parameters, we
compose a dataset of 44K fine-grained masks from several sources. HQ-SAM is
only trained on the introduced detaset of 44k masks, which takes only 4 hours
on 8 GPUs. We show the efficacy of HQ-SAM in a suite of 9 diverse segmentation
datasets across different downstream tasks, where 7 out of them are evaluated
in a zero-shot transfer protocol. Our code and models will be released at
https://github.com/SysCV/SAM-HQ.
HQ-SAM enhances the segmentation capabilities of SAM by introducing a learnable output token that leverages additional Vit features, improving mask quality with minimal computational overhead.
Segment Anything Model SAM zero-shot capabilities promptable design mask prediction mask decoder learnable High-Quality Output Token ViT features fine-grained masks zero-shot transfer