title: SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a
    Training-Free Memory Tree
summary: The Segment Anything Model 2 (SAM 2) has emerged as a powerful foundation
  model for object segmentation in both images and videos, paving the way for
  various downstream video applications. The crucial design of SAM 2 for video
  segmentation is its memory module, which prompts object-aware memories from
  previous frames for current frame prediction. However, its greedy-selection
  memory design suffers from the "error accumulation" problem, where an errored
  or missed mask will cascade and influence the segmentation of the subsequent
  frames, which limits the performance of SAM 2 toward complex long-term videos.
  To this end, we introduce SAM2Long, an improved training-free video object
  segmentation strategy, which considers the segmentation uncertainty within each
  frame and chooses the video-level optimal results from multiple segmentation
  pathways in a constrained tree search manner. In practice, we maintain a fixed
  number of segmentation pathways throughout the video. For each frame, multiple
  masks are proposed based on the existing pathways, creating various candidate
  branches. We then select the same fixed number of branches with higher
  cumulative scores as the new pathways for the next frame. After processing the
  final frame, the pathway with the highest cumulative score is chosen as the
  final segmentation result. Benefiting from its heuristic search design,
  SAM2Long is robust toward occlusions and object reappearances, and can
  effectively segment and track objects for complex long-term videos. Notably,
  SAM2Long achieves an average improvement of 3.0 points across all 24
  head-to-head comparisons, with gains of up to 5.3 points in J&F on long-term
  video object segmentation benchmarks such as SA-V and LVOS. The code is
  released at https://github.com/Mark12Ding/SAM2Long.
paper_authors._id: 67172a0c38b3deaf184d8a5c
paper_authors.name: Shuangrui Ding
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T08:21:34.470Z
paper_authors._id: 65a7c0335e79abfa2ec30c52
paper_authors.avatarUrl: /avatars/2f62f83f9c5c4cc9444571f067cd85b7.svg
paper_authors.fullname: Shuangrui Ding
paper_authors.type: user
paper_authors.user: Mar2Ding
paper_authors._id: 67172a0c38b3deaf184d8a5d
paper_authors.name: Rui Qian
paper_authors._id: 67172a0c38b3deaf184d8a5e
paper_authors.name: Xiaoyi Dong
paper_authors._id: 67172a0c38b3deaf184d8a5f
paper_authors.name: Pan Zhang
paper_authors._id: 67172a0c38b3deaf184d8a60
paper_authors.name: Yuhang Zang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T07:56:52.321Z
paper_authors._id: 63859cf3b2906edaf83af9f0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/iUQm5FAomzqYi6fkqIn9F.jpeg
paper_authors.fullname: Yuhang Zang
paper_authors.type: user
paper_authors.user: yuhangzang
paper_authors._id: 67172a0c38b3deaf184d8a61
paper_authors.name: Yuhang Cao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T08:25:55.933Z
paper_authors._id: 65000bef18830fabea469fdd
paper_authors.avatarUrl: /avatars/b320c77dfad039d9f9c54127f610d44f.svg
paper_authors.fullname: Cao Yuhang
paper_authors.type: user
paper_authors.user: yhcao
paper_authors._id: 67172a0c38b3deaf184d8a62
paper_authors.name: Yuwei Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T08:26:51.123Z
paper_authors._id: 6371f83d5ffa922d638ef486
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6371f83d5ffa922d638ef486/UzWl3xhVvtol3Rz1mjQ9t.jpeg
paper_authors.fullname: Yuwei Guo
paper_authors.type: user
paper_authors.user: guoyww
paper_authors._id: 67172a0c38b3deaf184d8a63
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T08:26:05.934Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_authors._id: 67172a0c38b3deaf184d8a64
paper_authors.name: Jiaqi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T07:56:54.146Z
paper_authors._id: 64b4eec4faa3181a5eab9c46
paper_authors.avatarUrl: /avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
paper_authors.fullname: Jiaqi Wang
paper_authors.type: user
paper_authors.user: myownskyW7
paper_title: SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a
    Training-Free Memory Tree
paper_summary: The Segment Anything Model 2 (SAM 2) has emerged as a powerful foundation
  model for object segmentation in both images and videos, paving the way for
  various downstream video applications. The crucial design of SAM 2 for video
  segmentation is its memory module, which prompts object-aware memories from
  previous frames for current frame prediction. However, its greedy-selection
  memory design suffers from the "error accumulation" problem, where an errored
  or missed mask will cascade and influence the segmentation of the subsequent
  frames, which limits the performance of SAM 2 toward complex long-term videos.
  To this end, we introduce SAM2Long, an improved training-free video object
  segmentation strategy, which considers the segmentation uncertainty within each
  frame and chooses the video-level optimal results from multiple segmentation
  pathways in a constrained tree search manner. In practice, we maintain a fixed
  number of segmentation pathways throughout the video. For each frame, multiple
  masks are proposed based on the existing pathways, creating various candidate
  branches. We then select the same fixed number of branches with higher
  cumulative scores as the new pathways for the next frame. After processing the
  final frame, the pathway with the highest cumulative score is chosen as the
  final segmentation result. Benefiting from its heuristic search design,
  SAM2Long is robust toward occlusions and object reappearances, and can
  effectively segment and track objects for complex long-term videos. Notably,
  SAM2Long achieves an average improvement of 3.0 points across all 24
  head-to-head comparisons, with gains of up to 5.3 points in J&F on long-term
  video object segmentation benchmarks such as SA-V and LVOS. The code is
  released at https://github.com/Mark12Ding/SAM2Long.
paper_ai_keywords: memory module object-aware memories greedy-selection error accumulation video object segmentation segmentation uncertainty constrained tree search segmentation pathways cumulative scores occlusions object reappearances