Surgical SAM 2: Real-time Segment Anything in Surgical Video by
  Efficient Frame Pruning
Surgical video segmentation is a critical task in computer-assisted surgery
and is vital for enhancing surgical quality and patient outcomes. Recently, the
Segment Anything Model 2 (SAM2) framework has shown superior advancements in
image and video segmentation. However, SAM2 struggles with efficiency due to
the high computational demands of processing high-resolution images and complex
and long-range temporal dynamics in surgical videos. To address these
challenges, we introduce Surgical SAM 2 (SurgSAM-2), an advanced model to
utilize SAM2 with an Efficient Frame Pruning (EFP) mechanism, to facilitate
real-time surgical video segmentation. The EFP mechanism dynamically manages
the memory bank by selectively retaining only the most informative frames,
reducing memory usage and computational cost while maintaining high
segmentation accuracy. Our extensive experiments demonstrate that SurgSAM-2
significantly improves both efficiency and segmentation accuracy compared to
the vanilla SAM2. Remarkably, SurgSAM-2 achieves a 3times FPS compared with
SAM2, while also delivering state-of-the-art performance after fine-tuning with
lower-resolution data. These advancements establish SurgSAM-2 as a leading
model for surgical video analysis, making real-time surgical video segmentation
in resource-constrained environments a feasible reality.
66bf77162777c05070c00057
Haofeng Liu
claimed_verified
2024-08-16T21:03:00.126Z
66bf75432777c05070bf49dc
/avatars/e18c2f6f5b1c37873dc181de4f255a8b.svg
Haofeng Liu
user
HeverLaw
66bf77162777c05070c00058
Erli Zhang
claimed_verified
2024-08-19T22:13:29.305Z
639fd77cbeb95d698de6671e
https://cdn-avatars.huggingface.co/v1/production/uploads/639fd77cbeb95d698de6671e/nP7IkgzDa2NhyP5K2CTMi.jpeg
Zhang Erli
user
KarlZEL
66bf77162777c05070c00059
Junde Wu
claimed_verified
2024-08-26T07:15:56.811Z
6317257fc92fd6fee317ff7c
/avatars/2f460a2f28562c987becb2acad8d93e7.svg
Junde Wu
user
morson
66bf77162777c05070c0005a
Mingxuan Hong
66bf77162777c05070c0005b
Yueming Jin
Surgical SAM 2: Real-time Segment Anything in Surgical Video by
  Efficient Frame Pruning
Surgical video segmentation is a critical task in computer-assisted surgery
and is vital for enhancing surgical quality and patient outcomes. Recently, the
Segment Anything Model 2 (SAM2) framework has shown superior advancements in
image and video segmentation. However, SAM2 struggles with efficiency due to
the high computational demands of processing high-resolution images and complex
and long-range temporal dynamics in surgical videos. To address these
challenges, we introduce Surgical SAM 2 (SurgSAM-2), an advanced model to
utilize SAM2 with an Efficient Frame Pruning (EFP) mechanism, to facilitate
real-time surgical video segmentation. The EFP mechanism dynamically manages
the memory bank by selectively retaining only the most informative frames,
reducing memory usage and computational cost while maintaining high
segmentation accuracy. Our extensive experiments demonstrate that SurgSAM-2
significantly improves both efficiency and segmentation accuracy compared to
the vanilla SAM2. Remarkably, SurgSAM-2 achieves a 3times FPS compared with
SAM2, while also delivering state-of-the-art performance after fine-tuning with
lower-resolution data. These advancements establish SurgSAM-2 as a leading
model for surgical video analysis, making real-time surgical video segmentation
in resource-constrained environments a feasible reality.
SurgSAM-2 enhances surgical video segmentation using an Efficient Frame Pruning mechanism, improving real-time performance and accuracy compared to SAM2.
Segment Anything Model 2 (SAM2) Surgical SAM 2 (SurgSAM-2) Efficient Frame Pruning (EFP) memory bank real-time surgical video segmentation