title: Personalize Segment Anything Model with One Shot
summary: Driven by large-data pre-training, Segment Anything Model (SAM) has been
  demonstrated as a powerful and promptable framework, revolutionizing the
  segmentation models. Despite the generality, customizing SAM for specific
  visual concepts without man-powered prompting is under explored, e.g.,
  automatically segmenting your pet dog in different images. In this paper, we
  propose a training-free Personalization approach for SAM, termed as PerSAM.
  Given only a single image with a reference mask, PerSAM first localizes the
  target concept by a location prior, and segments it within other images or
  videos via three techniques: target-guided attention, target-semantic
  prompting, and cascaded post-refinement. In this way, we effectively adapt SAM
  for private use without any training. To further alleviate the mask ambiguity,
  we present an efficient one-shot fine-tuning variant, PerSAM-F. Freezing the
  entire SAM, we introduce two learnable weights for multi-scale masks, only
  training 2 parameters within 10 seconds for improved performance. To
  demonstrate our efficacy, we construct a new segmentation dataset, PerSeg, for
  personalized evaluation, and test our methods on video object segmentation with
  competitive performance. Besides, our approach can also enhance DreamBooth to
  personalize Stable Diffusion for text-to-image generation, which discards the
  background disturbance for better target appearance learning. Code is released
  at https://github.com/ZrrSkywalker/Personalize-SAM
paper_authors._id: 645453d1bb73c0a97e992f60
paper_authors.name: Renrui Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-22T08:41:31.306Z
paper_authors._id: 645b8b2687c79b6ec0bb3b7a
paper_authors.avatarUrl: /avatars/00a9db32a42dc950112bf2593bb109cb.svg
paper_authors.fullname: Renrui
paper_authors.type: user
paper_authors.user: ZrrSkywalker
paper_authors._id: 645453d1bb73c0a97e992f61
paper_authors.name: Zhengkai Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-08T09:27:08.683Z
paper_authors._id: 6371dda2ffc0489ed7d712c2
paper_authors.avatarUrl: /avatars/dcb80cae00535c22af99baf86cb70d93.svg
paper_authors.fullname: zkjiang
paper_authors.type: user
paper_authors.user: justin-zk
paper_authors._id: 645453d1bb73c0a97e992f62
paper_authors.name: Ziyu Guo
paper_authors._id: 645453d1bb73c0a97e992f63
paper_authors.name: Shilin Yan
paper_authors._id: 645453d1bb73c0a97e992f64
paper_authors.name: Junting Pan
paper_authors._id: 645453d1bb73c0a97e992f65
paper_authors.name: Hao Dong
paper_authors._id: 645453d1bb73c0a97e992f66
paper_authors.name: Peng Gao
paper_authors._id: 645453d1bb73c0a97e992f67
paper_authors.name: Hongsheng Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-05T10:51:15.583Z
paper_authors._id: 65c04e9c27a5fdca81abcbd9
paper_authors.avatarUrl: /avatars/12a155683c824fa23da4a9e2bed4f64e.svg
paper_authors.fullname: Hongsheng LI
paper_authors.type: user
paper_authors.user: hsli-cuhk
paper_title: Personalize Segment Anything Model with One Shot
paper_summary: Driven by large-data pre-training, Segment Anything Model (SAM) has been
  demonstrated as a powerful and promptable framework, revolutionizing the
  segmentation models. Despite the generality, customizing SAM for specific
  visual concepts without man-powered prompting is under explored, e.g.,
  automatically segmenting your pet dog in different images. In this paper, we
  propose a training-free Personalization approach for SAM, termed as PerSAM.
  Given only a single image with a reference mask, PerSAM first localizes the
  target concept by a location prior, and segments it within other images or
  videos via three techniques: target-guided attention, target-semantic
  prompting, and cascaded post-refinement. In this way, we effectively adapt SAM
  for private use without any training. To further alleviate the mask ambiguity,
  we present an efficient one-shot fine-tuning variant, PerSAM-F. Freezing the
  entire SAM, we introduce two learnable weights for multi-scale masks, only
  training 2 parameters within 10 seconds for improved performance. To
  demonstrate our efficacy, we construct a new segmentation dataset, PerSeg, for
  personalized evaluation, and test our methods on video object segmentation with
  competitive performance. Besides, our approach can also enhance DreamBooth to
  personalize Stable Diffusion for text-to-image generation, which discards the
  background disturbance for better target appearance learning. Code is released
  at https://github.com/ZrrSkywalker/Personalize-SAM
paper_ai_keywords: Segment Anything Model (SAM) localization location prior target-guided attention target-semantic prompting cascaded post-refinement PerSAM PerSAM-F one-shot fine-tuning learnable weights multi-scale masks PerSeg video object segmentation DreamBooth Stable Diffusion text-to-image generation