title: Faster Segment Anything: Towards Lightweight SAM for Mobile Applications
summary: Segment anything model (SAM) is a prompt-guided vision foundation model for
  cutting out the object of interest from its background. Since Meta research
  team released the SA project, SAM has attracted significant attention due to
  its impressive zero-shot transfer performance and high versatility of being
  compatible with other models for advanced vision applications like image
  editing with fine-grained control. Many of such use cases need to be run on
  resource-constraint edge devices, like mobile Apps. In this work, we aim to
  make SAM mobile-friendly by replacing the heavyweight image encoder with a
  lightweight one. A naive way to train such a new SAM as in the original SAM
  paper leads to unsatisfactory performance, especially when limited training
  sources are available. We find that this is mainly caused by the coupled
  optimization of the image encoder and mask decoder, motivated by which we
  propose decoupled distillation. Concretely, we distill the knowledge from the
  image encoder ViT-H in the original SAM to a lightweight image encoder, which
  can be automatically compatible with the mask decoder in the original SAM. The
  training can be completed on a single GPU within less than one day, and the
  resulting lightweight SAM is termed MobileSAM which is more than 60 times
  smaller yet performs on par with the original SAM. For inference speed,
  MobileSAM runs around 10ms per image: 8ms on the image encoder and 2ms on the
  mask decoder. With superior performance and a higher versatility, our MobileSAM
  is 7 times smaller and 4 times faster than the concurrent FastSAM, making it
  more suitable for mobile applications. The code for MobileSAM project is
  provided at https://github.com/ChaoningZhang/MobileSAM
paper_authors._id: 649a870096d5747b35e7535d
paper_authors.name: Chaoning Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-28T09:25:28.697Z
paper_authors._id: 649a9b7cef592b26c3d39eb5
paper_authors.avatarUrl: /avatars/390b7ae94de540852ad5ce46bfb148e7.svg
paper_authors.fullname: Zhang
paper_authors.type: user
paper_authors.user: Chaoning
paper_authors._id: 649a870096d5747b35e7535e
paper_authors.name: Dongshen Han
paper_authors._id: 649a870096d5747b35e7535f
paper_authors.name: Yu Qiao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-28T20:15:24.018Z
paper_authors._id: 649bfed69ff0096550981c90
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/DFABS5u1v2DQ0vZ82V501.jpeg
paper_authors.fullname: Yu Qiao
paper_authors.type: user
paper_authors.user: qiaoyu1002
paper_authors._id: 649a870096d5747b35e75360
paper_authors.name: Jung Uk Kim
paper_authors._id: 649a870096d5747b35e75361
paper_authors.name: Sung-Ho Bae
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-28T20:15:21.809Z
paper_authors._id: 649bf062f8183583a4f1dec6
paper_authors.avatarUrl: /avatars/4317ffc9c2af6739cd27fd80aa1285b3.svg
paper_authors.fullname: Sung Ho Bae
paper_authors.type: user
paper_authors.user: shbae84
paper_authors._id: 649a870096d5747b35e75362
paper_authors.name: Seungkyu Lee
paper_authors._id: 649a870096d5747b35e75363
paper_authors.name: Choong Seon Hong
paper_title: Faster Segment Anything: Towards Lightweight SAM for Mobile Applications
paper_summary: Segment anything model (SAM) is a prompt-guided vision foundation model for
  cutting out the object of interest from its background. Since Meta research
  team released the SA project, SAM has attracted significant attention due to
  its impressive zero-shot transfer performance and high versatility of being
  compatible with other models for advanced vision applications like image
  editing with fine-grained control. Many of such use cases need to be run on
  resource-constraint edge devices, like mobile Apps. In this work, we aim to
  make SAM mobile-friendly by replacing the heavyweight image encoder with a
  lightweight one. A naive way to train such a new SAM as in the original SAM
  paper leads to unsatisfactory performance, especially when limited training
  sources are available. We find that this is mainly caused by the coupled
  optimization of the image encoder and mask decoder, motivated by which we
  propose decoupled distillation. Concretely, we distill the knowledge from the
  image encoder ViT-H in the original SAM to a lightweight image encoder, which
  can be automatically compatible with the mask decoder in the original SAM. The
  training can be completed on a single GPU within less than one day, and the
  resulting lightweight SAM is termed MobileSAM which is more than 60 times
  smaller yet performs on par with the original SAM. For inference speed,
  MobileSAM runs around 10ms per image: 8ms on the image encoder and 2ms on the
  mask decoder. With superior performance and a higher versatility, our MobileSAM
  is 7 times smaller and 4 times faster than the concurrent FastSAM, making it
  more suitable for mobile applications. The code for MobileSAM project is
  provided at https://github.com/ChaoningZhang/MobileSAM
paper_ai_keywords: prompt-guided vision foundation model zero-shot transfer heavyweight image encoder lightweight image encoder coupled optimization decoupled distillation mask decoder image encoder ViT-H MobileSAM inference speed