title: DC-SAM: In-Context Segment Anything in Images and Videos via Dual
    Consistency
summary: Given a single labeled example, in-context segmentation aims to segment
  corresponding objects. This setting, known as one-shot segmentation in few-shot
  learning, explores the segmentation model's generalization ability and has been
  applied to various vision tasks, including scene understanding and image/video
  editing. While recent Segment Anything Models have achieved state-of-the-art
  results in interactive segmentation, these approaches are not directly
  applicable to in-context segmentation. In this work, we propose the Dual
  Consistency SAM (DC-SAM) method based on prompt-tuning to adapt SAM and SAM2
  for in-context segmentation of both images and videos. Our key insights are to
  enhance the features of the SAM's prompt encoder in segmentation by providing
  high-quality visual prompts. When generating a mask prior, we fuse the SAM
  features to better align the prompt encoder. Then, we design a cycle-consistent
  cross-attention on fused features and initial visual prompts. Next, a
  dual-branch design is provided by using the discriminative positive and
  negative prompts in the prompt encoder. Furthermore, we design a simple
  mask-tube training strategy to adopt our proposed dual consistency method into
  the mask tube. Although the proposed DC-SAM is primarily designed for images,
  it can be seamlessly extended to the video domain with the support of SAM2.
  Given the absence of in-context segmentation in the video domain, we manually
  curate and construct the first benchmark from existing video segmentation
  datasets, named In-Context Video Object Segmentation (IC-VOS), to better assess
  the in-context capability of the model. Extensive experiments demonstrate that
  our method achieves 55.5 (+1.4) mIoU on COCO-20i, 73.0 (+1.1) mIoU on
  PASCAL-5i, and a J&F score of 71.52 on the proposed IC-VOS benchmark. Our
  source code and benchmark are available at https://github.com/zaplm/DC-SAM.
paper_authors._id: 680afc5f2c4b584e1d786eee
paper_authors.name: Mengshi Qi
paper_authors._id: 680afc5f2c4b584e1d786eef
paper_authors.name: Pengfei Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-25T08:34:30.195Z
paper_authors._id: 66a8c8e4f5cda7b8690205ef
paper_authors.avatarUrl: /avatars/5c43b7b50aeb3d1459307334ddcd1d1b.svg
paper_authors.fullname: Pengfei Zhu
paper_authors.type: user
paper_authors.user: zaplm
paper_authors._id: 680afc5f2c4b584e1d786ef0
paper_authors.name: Xiangtai Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:42:09.686Z
paper_authors._id: 63958b4414513eaf9029ebf1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U1g5H071pWRswGAG9UTpo.png
paper_authors.fullname: Xiangtai Li
paper_authors.type: user
paper_authors.user: LXT
paper_authors._id: 680afc5f2c4b584e1d786ef1
paper_authors.name: Xiaoyang Bi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:42:15.469Z
paper_authors._id: 659054f889145cbc7cc97978
paper_authors.avatarUrl: /avatars/74b7db0e170123e0df5247d64b2fbaab.svg
paper_authors.fullname: Xiaoyang Bi
paper_authors.type: user
paper_authors.user: xybi
paper_authors._id: 680afc5f2c4b584e1d786ef2
paper_authors.name: Lu Qi
paper_authors._id: 680afc5f2c4b584e1d786ef3
paper_authors.name: Huadong Ma
paper_authors._id: 680afc5f2c4b584e1d786ef4
paper_authors.name: Ming-Hsuan Yang
paper_title: DC-SAM: In-Context Segment Anything in Images and Videos via Dual
    Consistency
paper_summary: Given a single labeled example, in-context segmentation aims to segment
  corresponding objects. This setting, known as one-shot segmentation in few-shot
  learning, explores the segmentation model's generalization ability and has been
  applied to various vision tasks, including scene understanding and image/video
  editing. While recent Segment Anything Models have achieved state-of-the-art
  results in interactive segmentation, these approaches are not directly
  applicable to in-context segmentation. In this work, we propose the Dual
  Consistency SAM (DC-SAM) method based on prompt-tuning to adapt SAM and SAM2
  for in-context segmentation of both images and videos. Our key insights are to
  enhance the features of the SAM's prompt encoder in segmentation by providing
  high-quality visual prompts. When generating a mask prior, we fuse the SAM
  features to better align the prompt encoder. Then, we design a cycle-consistent
  cross-attention on fused features and initial visual prompts. Next, a
  dual-branch design is provided by using the discriminative positive and
  negative prompts in the prompt encoder. Furthermore, we design a simple
  mask-tube training strategy to adopt our proposed dual consistency method into
  the mask tube. Although the proposed DC-SAM is primarily designed for images,
  it can be seamlessly extended to the video domain with the support of SAM2.
  Given the absence of in-context segmentation in the video domain, we manually
  curate and construct the first benchmark from existing video segmentation
  datasets, named In-Context Video Object Segmentation (IC-VOS), to better assess
  the in-context capability of the model. Extensive experiments demonstrate that
  our method achieves 55.5 (+1.4) mIoU on COCO-20i, 73.0 (+1.1) mIoU on
  PASCAL-5i, and a J&F score of 71.52 on the proposed IC-VOS benchmark. Our
  source code and benchmark are available at https://github.com/zaplm/DC-SAM.
paper_ai_summary: The Dual Consistency SAM (DC-SAM) method enhances prompt-based feature alignment and introduces dual-branch and cycle-consistent cross-attention to improve in-context segmentation for both images and videos, achieving state-of-the-art results.
paper_ai_keywords: prompt-tuning SAM SAM2 in-context segmentation mask prior prompt encoder visual prompts cycle-consistent cross-attention dual-branch design mask-tube training In-Context Video Object Segmentation (IC-VOS) mIoU J&F score