title: EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything
    Model
summary: Segment Anything Model (SAM) has attracted widespread attention for its
  superior interactive segmentation capabilities with visual prompts while
  lacking further exploration of text prompts. In this paper, we empirically
  investigate what text prompt encoders (e.g., CLIP or LLM) are good for adapting
  SAM for referring expression segmentation and introduce the Early
  Vision-language Fusion-based SAM (EVF-SAM). EVF-SAM is a simple yet effective
  referring segmentation method which exploits multimodal prompts (i.e., image
  and text) and comprises a pre-trained vision-language model to generate
  referring prompts and a SAM model for segmentation. Surprisingly, we observe
  that: (1) multimodal prompts and (2) vision-language models with early fusion
  (e.g., BEIT-3) are beneficial for prompting SAM for accurate referring
  segmentation. Our experiments show that the proposed EVF-SAM based on BEIT-3
  can obtain state-of-the-art performance on RefCOCO/+/g for referring expression
  segmentation and demonstrate the superiority of prompting SAM with early
  vision-language fusion. In addition, the proposed EVF-SAM with 1.32B parameters
  achieves remarkably higher performance while reducing nearly 82% of parameters
  compared to previous SAM methods based on large multimodal models.
paper_authors._id: 66824779d892ee6886af5da2
paper_authors.name: Yuxuan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-08T07:42:30.084Z
paper_authors._id: 656d3ecdc497edf0a7b00a57
paper_authors.avatarUrl: /avatars/7879257bf247203a6dcf099b4076aeb5.svg
paper_authors.fullname: YxZhang
paper_authors.type: user
paper_authors.user: YxZhang
paper_authors._id: 66824779d892ee6886af5da3
paper_authors.name: Tianheng Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-01T07:46:29.538Z
paper_authors._id: 646b3db131968a60a01e4cf5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646b3db131968a60a01e4cf5/DhfdqUYQaD1Qa8Svw996J.jpeg
paper_authors.fullname: Tianheng Cheng
paper_authors.type: user
paper_authors.user: wondervictor
paper_authors._id: 66824779d892ee6886af5da4
paper_authors.name: Rui Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-01T09:43:04.536Z
paper_authors._id: 655bcabecfe086853e2aa22b
paper_authors.avatarUrl: /avatars/7fd2e7a5ccbbf25643e349682d2f655f.svg
paper_authors.fullname: Rui Hu
paper_authors.type: user
paper_authors.user: RuiHu
paper_authors._id: 66824779d892ee6886af5da5
paper_authors.name: ei Liu
paper_authors._id: 66824779d892ee6886af5da6
paper_authors.name: Heng Liu
paper_authors._id: 66824779d892ee6886af5da7
paper_authors.name: Longjin Ran
paper_authors._id: 66824779d892ee6886af5da8
paper_authors.name: Xiaoxin Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-01T09:42:39.544Z
paper_authors._id: 65389a669c474315d7425f96
paper_authors.avatarUrl: /avatars/2fa3828ca489cfe1948129a0eccf264f.svg
paper_authors.fullname: chenxiaoxin
paper_authors.type: user
paper_authors.user: steelozazala
paper_authors._id: 66824779d892ee6886af5da9
paper_authors.name: Wenyu Liu
paper_authors._id: 66824779d892ee6886af5daa
paper_authors.name: Xinggang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-01T09:42:04.985Z
paper_authors._id: 62600de6d47e3dbae32ce1ce
paper_authors.avatarUrl: /avatars/a536417cfec6e10ac415091bd1829426.svg
paper_authors.fullname: Xinggang Wang
paper_authors.type: user
paper_authors.user: xinggangw
paper_title: EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything
    Model
paper_summary: Segment Anything Model (SAM) has attracted widespread attention for its
  superior interactive segmentation capabilities with visual prompts while
  lacking further exploration of text prompts. In this paper, we empirically
  investigate what text prompt encoders (e.g., CLIP or LLM) are good for adapting
  SAM for referring expression segmentation and introduce the Early
  Vision-language Fusion-based SAM (EVF-SAM). EVF-SAM is a simple yet effective
  referring segmentation method which exploits multimodal prompts (i.e., image
  and text) and comprises a pre-trained vision-language model to generate
  referring prompts and a SAM model for segmentation. Surprisingly, we observe
  that: (1) multimodal prompts and (2) vision-language models with early fusion
  (e.g., BEIT-3) are beneficial for prompting SAM for accurate referring
  segmentation. Our experiments show that the proposed EVF-SAM based on BEIT-3
  can obtain state-of-the-art performance on RefCOCO/+/g for referring expression
  segmentation and demonstrate the superiority of prompting SAM with early
  vision-language fusion. In addition, the proposed EVF-SAM with 1.32B parameters
  achieves remarkably higher performance while reducing nearly 82% of parameters
  compared to previous SAM methods based on large multimodal models.
paper_ai_keywords: Segment Anything Model (SAM) referring expression segmentation text prompt encoders CLIP LLM Early Vision-language Fusion-based SAM (EVF-SAM) multimodal prompts pre-trained vision-language model BEIT-3 RefCOCO/+/g vision-language fusion