SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a
  Training-Free Memory Tree
The Segment Anything Model 2 (SAM 2) has emerged as a powerful foundation
model for object segmentation in both images and videos, paving the way for
various downstream video applications. The crucial design of SAM 2 for video
segmentation is its memory module, which prompts object-aware memories from
previous frames for current frame prediction. However, its greedy-selection
memory design suffers from the "error accumulation" problem, where an errored
or missed mask will cascade and influence the segmentation of the subsequent
frames, which limits the performance of SAM 2 toward complex long-term videos.
To this end, we introduce SAM2Long, an improved training-free video object
segmentation strategy, which considers the segmentation uncertainty within each
frame and chooses the video-level optimal results from multiple segmentation
pathways in a constrained tree search manner. In practice, we maintain a fixed
number of segmentation pathways throughout the video. For each frame, multiple
masks are proposed based on the existing pathways, creating various candidate
branches. We then select the same fixed number of branches with higher
cumulative scores as the new pathways for the next frame. After processing the
final frame, the pathway with the highest cumulative score is chosen as the
final segmentation result. Benefiting from its heuristic search design,
SAM2Long is robust toward occlusions and object reappearances, and can
effectively segment and track objects for complex long-term videos. Notably,
SAM2Long achieves an average improvement of 3.0 points across all 24
head-to-head comparisons, with gains of up to 5.3 points in J&F on long-term
video object segmentation benchmarks such as SA-V and LVOS. The code is
released at https://github.com/Mark12Ding/SAM2Long.
67172a0c38b3deaf184d8a5c
Shuangrui Ding
admin_assigned
2024-10-22T08:21:34.470Z
65a7c0335e79abfa2ec30c52
/avatars/2f62f83f9c5c4cc9444571f067cd85b7.svg
Shuangrui Ding
user
Mar2Ding
67172a0c38b3deaf184d8a5d
Rui Qian
67172a0c38b3deaf184d8a5e
Xiaoyi Dong
67172a0c38b3deaf184d8a5f
Pan Zhang
67172a0c38b3deaf184d8a60
Yuhang Zang
claimed_verified
2024-10-22T07:56:52.321Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/mCgynEtoXdILvzyoPexii.png
Yuhang Zang
user
yuhangzang
67172a0c38b3deaf184d8a61
Yuhang Cao
admin_assigned
2024-10-22T08:25:55.933Z
65000bef18830fabea469fdd
/avatars/b320c77dfad039d9f9c54127f610d44f.svg
Cao Yuhang
user
yhcao
67172a0c38b3deaf184d8a62
Yuwei Guo
admin_assigned
2024-10-22T08:26:51.123Z
6371f83d5ffa922d638ef486
https://cdn-avatars.huggingface.co/v1/production/uploads/6371f83d5ffa922d638ef486/UzWl3xhVvtol3Rz1mjQ9t.jpeg
Yuwei Guo
user
guoyww
67172a0c38b3deaf184d8a63
Dahua Lin
admin_assigned
2024-10-22T08:26:05.934Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
67172a0c38b3deaf184d8a64
Jiaqi Wang
claimed_verified
2024-10-22T07:56:54.146Z
64b4eec4faa3181a5eab9c46
/avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
Jiaqi Wang
user
myownskyW7
SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a
  Training-Free Memory Tree
The Segment Anything Model 2 (SAM 2) has emerged as a powerful foundation
model for object segmentation in both images and videos, paving the way for
various downstream video applications. The crucial design of SAM 2 for video
segmentation is its memory module, which prompts object-aware memories from
previous frames for current frame prediction. However, its greedy-selection
memory design suffers from the "error accumulation" problem, where an errored
or missed mask will cascade and influence the segmentation of the subsequent
frames, which limits the performance of SAM 2 toward complex long-term videos.
To this end, we introduce SAM2Long, an improved training-free video object
segmentation strategy, which considers the segmentation uncertainty within each
frame and chooses the video-level optimal results from multiple segmentation
pathways in a constrained tree search manner. In practice, we maintain a fixed
number of segmentation pathways throughout the video. For each frame, multiple
masks are proposed based on the existing pathways, creating various candidate
branches. We then select the same fixed number of branches with higher
cumulative scores as the new pathways for the next frame. After processing the
final frame, the pathway with the highest cumulative score is chosen as the
final segmentation result. Benefiting from its heuristic search design,
SAM2Long is robust toward occlusions and object reappearances, and can
effectively segment and track objects for complex long-term videos. Notably,
SAM2Long achieves an average improvement of 3.0 points across all 24
head-to-head comparisons, with gains of up to 5.3 points in J&F on long-term
video object segmentation benchmarks such as SA-V and LVOS. The code is
released at https://github.com/Mark12Ding/SAM2Long.
SAM2Long enhances SAM 2's video segmentation by using a memory module and constrained tree search to improve accuracy in complex, long-term videos.
memory module video segmentation error accumulation segmentation uncertainty constrained tree search heuristic search object tracking long-term video object segmentation J&F SA-V LVOS