title: Unifying Segment Anything in Microscopy with Multimodal Large Language
    Model
summary: Accurate segmentation of regions of interest in biomedical images holds
  substantial value in image analysis. Although several foundation models for
  biomedical segmentation have currently achieved excellent performance on
  certain datasets, they typically demonstrate sub-optimal performance on unseen
  domain data. We owe the deficiency to lack of vision-language knowledge before
  segmentation. Multimodal Large Language Models (MLLMs) bring outstanding
  understanding and reasoning capabilities to multimodal tasks, which inspires us
  to leverage MLLMs to inject Vision-Language Knowledge (VLK), thereby enabling
  vision models to demonstrate superior generalization capabilities on
  cross-domain datasets. In this paper, we propose using MLLMs to guide SAM in
  learning microscopy crose-domain data, unifying Segment Anything in Microscopy,
  named uLLSAM. Specifically, we propose the Vision-Language Semantic Alignment
  (VLSA) module, which injects VLK into Segment Anything Model (SAM). We find
  that after SAM receives global VLK prompts, its performance improves
  significantly, but there are deficiencies in boundary contour perception.
  Therefore, we further propose Semantic Boundary Regularization (SBR) to prompt
  SAM. Our method achieves performance improvements of 7.71% in Dice and 12.10%
  in SA across 9 in-domain microscopy datasets, achieving state-of-the-art
  performance. Our method also demonstrates improvements of 6.79% in Dice and
  10.08% in SA across 10 out-ofdomain datasets, exhibiting strong generalization
  capabilities. Code is available at https://github.com/ieellee/uLLSAM.
paper_authors._id: 682bb1d99476a97d3b983da0
paper_authors.name: Manyu Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-20T07:22:02.593Z
paper_authors._id: 66e01d290b0271487147fdc9
paper_authors.avatarUrl: /avatars/2dfb01fba8b781a15555f52f8b92d06c.svg
paper_authors.fullname: Manyu Li
paper_authors.type: user
paper_authors.user: ieellee
paper_authors._id: 682bb1d99476a97d3b983da1
paper_authors.name: Ruian He
paper_authors._id: 682bb1d99476a97d3b983da2
paper_authors.name: Zixian Zhang
paper_authors._id: 682bb1d99476a97d3b983da3
paper_authors.name: Weimin Tan
paper_authors._id: 682bb1d99476a97d3b983da4
paper_authors.name: Bo Yan
paper_title: Unifying Segment Anything in Microscopy with Multimodal Large Language
    Model
paper_summary: Accurate segmentation of regions of interest in biomedical images holds
  substantial value in image analysis. Although several foundation models for
  biomedical segmentation have currently achieved excellent performance on
  certain datasets, they typically demonstrate sub-optimal performance on unseen
  domain data. We owe the deficiency to lack of vision-language knowledge before
  segmentation. Multimodal Large Language Models (MLLMs) bring outstanding
  understanding and reasoning capabilities to multimodal tasks, which inspires us
  to leverage MLLMs to inject Vision-Language Knowledge (VLK), thereby enabling
  vision models to demonstrate superior generalization capabilities on
  cross-domain datasets. In this paper, we propose using MLLMs to guide SAM in
  learning microscopy crose-domain data, unifying Segment Anything in Microscopy,
  named uLLSAM. Specifically, we propose the Vision-Language Semantic Alignment
  (VLSA) module, which injects VLK into Segment Anything Model (SAM). We find
  that after SAM receives global VLK prompts, its performance improves
  significantly, but there are deficiencies in boundary contour perception.
  Therefore, we further propose Semantic Boundary Regularization (SBR) to prompt
  SAM. Our method achieves performance improvements of 7.71% in Dice and 12.10%
  in SA across 9 in-domain microscopy datasets, achieving state-of-the-art
  performance. Our method also demonstrates improvements of 6.79% in Dice and
  10.08% in SA across 10 out-ofdomain datasets, exhibiting strong generalization
  capabilities. Code is available at https://github.com/ieellee/uLLSAM.
paper_ai_summary: Injecting Vision-Language Knowledge using MLLMs enhances SAM for better performance and generalization across in-domain and out-of-domain microscopy datasets.
paper_ai_keywords: Multimodal Large Language Models MLLMs Vision-Language Knowledge VLK Segment Anything Model SAM Vision-Language Semantic Alignment VLSA Semantic Boundary Regularization SBR Dice SA