title: Mamba as a Bridge: Where Vision Foundation Models Meet Vision Language
    Models for Domain-Generalized Semantic Segmentation
summary: Vision Foundation Models (VFMs) and Vision-Language Models (VLMs) have gained
  traction in Domain Generalized Semantic Segmentation (DGSS) due to their strong
  generalization capabilities. However, existing DGSS methods often rely
  exclusively on either VFMs or VLMs, overlooking their complementary strengths.
  VFMs (e.g., DINOv2) excel at capturing fine-grained features, while VLMs (e.g.,
  CLIP) provide robust text alignment but struggle with coarse granularity.
  Despite their complementary strengths, effectively integrating VFMs and VLMs
  with attention mechanisms is challenging, as the increased patch tokens
  complicate long-sequence modeling. To address this, we propose MFuser, a novel
  Mamba-based fusion framework that efficiently combines the strengths of VFMs
  and VLMs while maintaining linear scalability in sequence length. MFuser
  consists of two key components: MVFuser, which acts as a co-adapter to jointly
  fine-tune the two models by capturing both sequential and spatial dynamics; and
  MTEnhancer, a hybrid attention-Mamba module that refines text embeddings by
  incorporating image priors. Our approach achieves precise feature locality and
  strong text alignment without incurring significant computational overhead.
  Extensive experiments demonstrate that MFuser significantly outperforms
  state-of-the-art DGSS methods, achieving 68.20 mIoU on synthetic-to-real and
  71.87 mIoU on real-to-real benchmarks. The code is available at
  https://github.com/devinxzhang/MFuser.
paper_authors._id: 67f3f25019592b36b6d8b30c
paper_authors.name: Xin Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-08T06:52:23.735Z
paper_authors._id: 6436290e76dbfd731bcf1f55
paper_authors.avatarUrl: /avatars/63af84ca382d799c4a000594c994d4bb.svg
paper_authors.fullname: Xin Zhang
paper_authors.type: user
paper_authors.user: XinNUS
paper_authors._id: 67f3f25019592b36b6d8b30d
paper_authors.name: Robby T. Tan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-08T10:39:06.222Z
paper_authors._id: 67c88df8eeb202fffea14efd
paper_authors.avatarUrl: /avatars/915f516389c99f03495b98a41fca704b.svg
paper_authors.fullname: robbytan
paper_authors.type: user
paper_authors.user: robbytan
paper_title: Mamba as a Bridge: Where Vision Foundation Models Meet Vision Language
    Models for Domain-Generalized Semantic Segmentation
paper_summary: Vision Foundation Models (VFMs) and Vision-Language Models (VLMs) have gained
  traction in Domain Generalized Semantic Segmentation (DGSS) due to their strong
  generalization capabilities. However, existing DGSS methods often rely
  exclusively on either VFMs or VLMs, overlooking their complementary strengths.
  VFMs (e.g., DINOv2) excel at capturing fine-grained features, while VLMs (e.g.,
  CLIP) provide robust text alignment but struggle with coarse granularity.
  Despite their complementary strengths, effectively integrating VFMs and VLMs
  with attention mechanisms is challenging, as the increased patch tokens
  complicate long-sequence modeling. To address this, we propose MFuser, a novel
  Mamba-based fusion framework that efficiently combines the strengths of VFMs
  and VLMs while maintaining linear scalability in sequence length. MFuser
  consists of two key components: MVFuser, which acts as a co-adapter to jointly
  fine-tune the two models by capturing both sequential and spatial dynamics; and
  MTEnhancer, a hybrid attention-Mamba module that refines text embeddings by
  incorporating image priors. Our approach achieves precise feature locality and
  strong text alignment without incurring significant computational overhead.
  Extensive experiments demonstrate that MFuser significantly outperforms
  state-of-the-art DGSS methods, achieving 68.20 mIoU on synthetic-to-real and
  71.87 mIoU on real-to-real benchmarks. The code is available at
  https://github.com/devinxzhang/MFuser.
paper_ai_summary: MFuser combines Vision Foundation Models and Vision-Language Models using Mamba-based fusion for enhanced Domain Generalized Semantic Segmentation, achieving high performance with efficient scalability.
paper_ai_keywords: Vision Foundation Models Vision-Language Models Domain Generalized Semantic Segmentation DINOv2 CLIP patch tokens long-sequence modeling Mamba-based fusion MFuser MVFuser MTEnhancer hybrid attention-Mamba feature locality text alignment mIoU