title: Segment Anything Meets Point Tracking
summary: The Segment Anything Model (SAM) has established itself as a powerful
  zero-shot image segmentation model, employing interactive prompts such as
  points to generate masks. This paper presents SAM-PT, a method extending SAM's
  capability to tracking and segmenting anything in dynamic videos. SAM-PT
  leverages robust and sparse point selection and propagation techniques for mask
  generation, demonstrating that a SAM-based segmentation tracker can yield
  strong zero-shot performance across popular video object segmentation
  benchmarks, including DAVIS, YouTube-VOS, and MOSE. Compared to traditional
  object-centric mask propagation strategies, we uniquely use point propagation
  to exploit local structure information that is agnostic to object semantics. We
  highlight the merits of point-based tracking through direct evaluation on the
  zero-shot open-world Unidentified Video Objects (UVO) benchmark. To further
  enhance our approach, we utilize K-Medoids clustering for point initialization
  and track both positive and negative points to clearly distinguish the target
  object. We also employ multiple mask decoding passes for mask refinement and
  devise a point re-initialization strategy to improve tracking accuracy. Our
  code integrates different point trackers and video segmentation benchmarks and
  will be released at https://github.com/SysCV/sam-pt.
paper_authors._id: 64a3a60b7068337139e52c65
paper_authors.name: Frano Rajič
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-04T19:16:26.536Z
paper_authors._id: 64a40013aab6329074397dfe
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/I13uQeHFl9ynuS_Sur6rY.png
paper_authors.fullname: Frano Rajič
paper_authors.type: user
paper_authors.user: m43
paper_authors._id: 64a3a60b7068337139e52c66
paper_authors.name: Lei Ke
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-04T19:16:30.734Z
paper_authors._id: 6258a6455ea3a0a9b6de3f22
paper_authors.avatarUrl: /avatars/6eeed72a97fb24465e5e65583fbe50cf.svg
paper_authors.fullname: Lei Ke
paper_authors.type: user
paper_authors.user: lkeab
paper_authors._id: 64a3a60b7068337139e52c67
paper_authors.name: Yu-Wing Tai
paper_authors._id: 64a3a60b7068337139e52c68
paper_authors.name: Chi-Keung Tang
paper_authors._id: 64a3a60b7068337139e52c69
paper_authors.name: Martin Danelljan
paper_authors._id: 64a3a60b7068337139e52c6a
paper_authors.name: Fisher Yu
paper_title: Segment Anything Meets Point Tracking
paper_summary: The Segment Anything Model (SAM) has established itself as a powerful
  zero-shot image segmentation model, employing interactive prompts such as
  points to generate masks. This paper presents SAM-PT, a method extending SAM's
  capability to tracking and segmenting anything in dynamic videos. SAM-PT
  leverages robust and sparse point selection and propagation techniques for mask
  generation, demonstrating that a SAM-based segmentation tracker can yield
  strong zero-shot performance across popular video object segmentation
  benchmarks, including DAVIS, YouTube-VOS, and MOSE. Compared to traditional
  object-centric mask propagation strategies, we uniquely use point propagation
  to exploit local structure information that is agnostic to object semantics. We
  highlight the merits of point-based tracking through direct evaluation on the
  zero-shot open-world Unidentified Video Objects (UVO) benchmark. To further
  enhance our approach, we utilize K-Medoids clustering for point initialization
  and track both positive and negative points to clearly distinguish the target
  object. We also employ multiple mask decoding passes for mask refinement and
  devise a point re-initialization strategy to improve tracking accuracy. Our
  code integrates different point trackers and video segmentation benchmarks and
  will be released at https://github.com/SysCV/sam-pt.
paper_ai_keywords: zero-shot image segmentation interactive prompts mask generation zero-shot performance video object segmentation DAVIS YouTube-VOS MOSE point selection point propagation local structure information object semantics point-based tracking zero-shot open-world Unidentified Video Objects (UVO) K-Medoids clustering mask refinement point re-initialization point trackers video segmentation benchmarks