title: MobileSAMv2: Faster Segment Anything to Everything
summary: Segment anything model (SAM) addresses two practical yet challenging
  segmentation tasks: segment anything (SegAny), which utilizes a
  certain point to predict the mask for a single object of interest, and
  segment everything (SegEvery), which predicts the masks for all
  objects on the image. What makes SegAny slow for SAM is its heavyweight image
  encoder, which has been addressed by MobileSAM via decoupled knowledge
  distillation. The efficiency bottleneck of SegEvery with SAM, however, lies in
  its mask decoder because it needs to first generate numerous masks with
  redundant grid-search prompts and then perform filtering to obtain the final
  valid masks. We propose to improve its efficiency by directly generating the
  final masks with only valid prompts, which can be obtained through object
  discovery. Our proposed approach not only helps reduce the total time on the
  mask decoder by at least 16 times but also achieves superior performance.
  Specifically, our approach yields an average performance boost of 3.6\% (42.5\%
  v.s. 38.9\%) for zero-shot object proposal on the LVIS dataset with
  the mask AR@K metric. Qualitative results show that our approach generates
  fine-grained masks while avoiding over-segmenting things. This project
  targeting faster SegEvery than the original SAM is termed MobileSAMv2 to
  differentiate from MobileSAM which targets faster SegAny. Moreover, we
  demonstrate that our new prompt sampling is also compatible with the distilled
  image encoders in MobileSAM, contributing to a unified framework for efficient
  SegAny and SegEvery. The code is available at the same link as MobileSAM
  Project
  https://github.com/ChaoningZhang/MobileSAM{red{https://github.com/ChaoningZhang/MobileSAM}}.
  abstract
paper_authors._id: 657fd85d9a9e347d4bfbda87
paper_authors.name: Chaoning Zhang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-12-18T05:27:58.662Z
paper_authors._id: 649a9b7cef592b26c3d39eb5
paper_authors.avatarUrl: /avatars/390b7ae94de540852ad5ce46bfb148e7.svg
paper_authors.fullname: Zhang
paper_authors.type: user
paper_authors.user: Chaoning
paper_authors._id: 657fd85d9a9e347d4bfbda88
paper_authors.name: Dongshen Han
paper_authors._id: 657fd85d9a9e347d4bfbda89
paper_authors.name: Sheng Zheng
paper_authors._id: 657fd85d9a9e347d4bfbda8a
paper_authors.name: Jinwoo Choi
paper_authors._id: 657fd85d9a9e347d4bfbda8b
paper_authors.name: Tae-Ho Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-18T11:06:40.592Z
paper_authors._id: 64cccbe8c7f30fbf7bbf5a6d
paper_authors.avatarUrl: /avatars/1c1f7c2da22c2afb3b352d446e80e5ef.svg
paper_authors.fullname: Tae-Ho Kim
paper_authors.type: user
paper_authors.user: ktho22
paper_authors._id: 657fd85d9a9e347d4bfbda8c
paper_authors.name: Choong Seon Hong
paper_title: MobileSAMv2: Faster Segment Anything to Everything
paper_summary: Segment anything model (SAM) addresses two practical yet challenging
  segmentation tasks: segment anything (SegAny), which utilizes a
  certain point to predict the mask for a single object of interest, and
  segment everything (SegEvery), which predicts the masks for all
  objects on the image. What makes SegAny slow for SAM is its heavyweight image
  encoder, which has been addressed by MobileSAM via decoupled knowledge
  distillation. The efficiency bottleneck of SegEvery with SAM, however, lies in
  its mask decoder because it needs to first generate numerous masks with
  redundant grid-search prompts and then perform filtering to obtain the final
  valid masks. We propose to improve its efficiency by directly generating the
  final masks with only valid prompts, which can be obtained through object
  discovery. Our proposed approach not only helps reduce the total time on the
  mask decoder by at least 16 times but also achieves superior performance.
  Specifically, our approach yields an average performance boost of 3.6\% (42.5\%
  v.s. 38.9\%) for zero-shot object proposal on the LVIS dataset with
  the mask AR@K metric. Qualitative results show that our approach generates
  fine-grained masks while avoiding over-segmenting things. This project
  targeting faster SegEvery than the original SAM is termed MobileSAMv2 to
  differentiate from MobileSAM which targets faster SegAny. Moreover, we
  demonstrate that our new prompt sampling is also compatible with the distilled
  image encoders in MobileSAM, contributing to a unified framework for efficient
  SegAny and SegEvery. The code is available at the same link as MobileSAM
  Project
  https://github.com/ChaoningZhang/MobileSAM{red{https://github.com/ChaoningZhang/MobileSAM}}.
  abstract
paper_ai_summary: A new approach improves MobileSAM's efficiency for the SegEvery task by directly generating valid masks, boosting performance and reducing computation time.
paper_ai_keywords: segment anything segmentation tasks image encoder heavy image encoder decoupled knowledge distillation mask decoder mask generation object discovery prompt sampling LVIS dataset mask AR@$K$ metric zero-shot object proposal