ReSurgSAM2: Referring Segment Anything in Surgical Video via Credible
  Long-term Tracking
Surgical scene segmentation is critical in computer-assisted surgery and is
vital for enhancing surgical quality and patient outcomes. Recently, referring
surgical segmentation is emerging, given its advantage of providing surgeons
with an interactive experience to segment the target object. However, existing
methods are limited by low efficiency and short-term tracking, hindering their
applicability in complex real-world surgical scenarios. In this paper, we
introduce ReSurgSAM2, a two-stage surgical referring segmentation framework
that leverages Segment Anything Model 2 to perform text-referred target
detection, followed by tracking with reliable initial frame identification and
diversity-driven long-term memory. For the detection stage, we propose a
cross-modal spatial-temporal Mamba to generate precise detection and
segmentation results. Based on these results, our credible initial frame
selection strategy identifies the reliable frame for the subsequent tracking.
Upon selecting the initial frame, our method transitions to the tracking stage,
where it incorporates a diversity-driven memory mechanism that maintains a
credible and diverse memory bank, ensuring consistent long-term tracking.
Extensive experiments demonstrate that ReSurgSAM2 achieves substantial
improvements in accuracy and efficiency compared to existing methods, operating
in real-time at 61.2 FPS. Our code and datasets will be available at
https://github.com/jinlab-imvr/ReSurgSAM2.
6825c47fbe0e3c3bfbf4d285
Haofeng Liu
claimed_verified
2025-05-16T07:12:36.358Z
66bf75432777c05070bf49dc
/avatars/e18c2f6f5b1c37873dc181de4f255a8b.svg
Haofeng Liu
user
HeverLaw
6825c47fbe0e3c3bfbf4d286
Mingqi Gao
6825c47fbe0e3c3bfbf4d287
Xuxiao Luo
6825c47fbe0e3c3bfbf4d288
Ziyue Wang
6825c47fbe0e3c3bfbf4d289
Guanyi Qin
6825c47fbe0e3c3bfbf4d28a
Junde Wu
admin_assigned
2025-05-16T14:20:21.901Z
6317257fc92fd6fee317ff7c
/avatars/2f460a2f28562c987becb2acad8d93e7.svg
Junde Wu
user
morson
6825c47fbe0e3c3bfbf4d28b
Yueming Jin
ReSurgSAM2: Referring Segment Anything in Surgical Video via Credible
  Long-term Tracking
Surgical scene segmentation is critical in computer-assisted surgery and is
vital for enhancing surgical quality and patient outcomes. Recently, referring
surgical segmentation is emerging, given its advantage of providing surgeons
with an interactive experience to segment the target object. However, existing
methods are limited by low efficiency and short-term tracking, hindering their
applicability in complex real-world surgical scenarios. In this paper, we
introduce ReSurgSAM2, a two-stage surgical referring segmentation framework
that leverages Segment Anything Model 2 to perform text-referred target
detection, followed by tracking with reliable initial frame identification and
diversity-driven long-term memory. For the detection stage, we propose a
cross-modal spatial-temporal Mamba to generate precise detection and
segmentation results. Based on these results, our credible initial frame
selection strategy identifies the reliable frame for the subsequent tracking.
Upon selecting the initial frame, our method transitions to the tracking stage,
where it incorporates a diversity-driven memory mechanism that maintains a
credible and diverse memory bank, ensuring consistent long-term tracking.
Extensive experiments demonstrate that ReSurgSAM2 achieves substantial
improvements in accuracy and efficiency compared to existing methods, operating
in real-time at 61.2 FPS. Our code and datasets will be available at
https://github.com/jinlab-imvr/ReSurgSAM2.
ReSurgSAM2 is a two-stage framework that improves surgical scene segmentation and tracking using Segment Anything Model 2, cross-modal spatial-temporal Mamba, and a diversity-driven memory mechanism.
ReSurgSAM2 Segment Anything Model 2 cross-modal spatial-temporal Mamba diversity-driven long-term memory