TinySAM: Pushing the Envelope for Efficient Segment Anything Model
Recently segment anything model (SAM) has shown powerful segmentation
capability and has drawn great attention in computer vision fields. Massive
following works have developed various applications based on the pretrained SAM
and achieved impressive performance on downstream vision tasks. However, SAM
consists of heavy architectures and requires massive computational capacity,
which hinders the further application of SAM on computation constrained edge
devices. To this end, in this paper we propose a framework to obtain a tiny
segment anything model (TinySAM) while maintaining the strong zero-shot
performance. We first propose a full-stage knowledge distillation method with
online hard prompt sampling strategy to distill a lightweight student model. We
also adapt the post-training quantization to the promptable segmentation task
and further reduce the computational cost. Moreover, a hierarchical segmenting
everything strategy is proposed to accelerate the everything inference by
2times with almost no performance degradation. With all these proposed
methods, our TinySAM leads to orders of magnitude computational reduction and
pushes the envelope for efficient segment anything task. Extensive experiments
on various zero-shot transfer tasks demonstrate the significantly advantageous
performance of our TinySAM against counterpart methods. Pre-trained models and
codes will be available at https://github.com/xinghaochen/TinySAM and
https://gitee.com/mindspore/models/tree/master/research/cv/TinySAM.
6584f07641dbedb146fb2ca0
Han Shu
6584f07641dbedb146fb2ca1
Wenshuo Li
6584f07641dbedb146fb2ca2
Yehui Tang
admin_assigned
2023-12-22T11:21:14.139Z
64d5deb154bb9eb704f83122
/avatars/86ce09bcca903319051e2307581a43f4.svg
Yehui Tang
user
tangyehui
6584f07641dbedb146fb2ca3
Yiman Zhang
admin_assigned
2023-12-22T11:21:20.400Z
64be15c98496ee0fb60e4c22
/avatars/09e0b747e43c3aadeaa8a715226ee0c0.svg
Yiman Zhang
user
YimanZhang
6584f07641dbedb146fb2ca4
Yihao Chen
6584f07641dbedb146fb2ca5
Houqiang Li
6584f07641dbedb146fb2ca6
Yunhe Wang
6584f07641dbedb146fb2ca7
Xinghao Chen
claimed_verified
2024-05-16T10:46:21.300Z
65878c774f9d2b955e025019
/avatars/c039c29e131a8cba96b38048da94971b.svg
Xinghao Chen
user
xinghaochen
TinySAM: Pushing the Envelope for Efficient Segment Anything Model
Recently segment anything model (SAM) has shown powerful segmentation
capability and has drawn great attention in computer vision fields. Massive
following works have developed various applications based on the pretrained SAM
and achieved impressive performance on downstream vision tasks. However, SAM
consists of heavy architectures and requires massive computational capacity,
which hinders the further application of SAM on computation constrained edge
devices. To this end, in this paper we propose a framework to obtain a tiny
segment anything model (TinySAM) while maintaining the strong zero-shot
performance. We first propose a full-stage knowledge distillation method with
online hard prompt sampling strategy to distill a lightweight student model. We
also adapt the post-training quantization to the promptable segmentation task
and further reduce the computational cost. Moreover, a hierarchical segmenting
everything strategy is proposed to accelerate the everything inference by
2times with almost no performance degradation. With all these proposed
methods, our TinySAM leads to orders of magnitude computational reduction and
pushes the envelope for efficient segment anything task. Extensive experiments
on various zero-shot transfer tasks demonstrate the significantly advantageous
performance of our TinySAM against counterpart methods. Pre-trained models and
codes will be available at https://github.com/xinghaochen/TinySAM and
https://gitee.com/mindspore/models/tree/master/research/cv/TinySAM.
TinySAM reduces computational cost while maintaining strong zero-shot segmentation performance through knowledge distillation, quantization, and hierarchical segmentation strategies.
knowledge distillation online hard prompt sampling post-training quantization hierarchical segmenting everything strategy zero-shot segmentation