title: Segment Anything with Multiple Modalities
summary: Robust and accurate segmentation of scenes has become one core functionality
  in various visual recognition and navigation tasks. This has inspired the
  recent development of Segment Anything Model (SAM), a foundation model for
  general mask segmentation. However, SAM is largely tailored for single-modal
  RGB images, limiting its applicability to multi-modal data captured with
  widely-adopted sensor suites, such as LiDAR plus RGB, depth plus RGB, thermal
  plus RGB, etc. We develop MM-SAM, an extension and expansion of SAM that
  supports cross-modal and multi-modal processing for robust and enhanced
  segmentation with different sensor suites. MM-SAM features two key designs,
  namely, unsupervised cross-modal transfer and weakly-supervised multi-modal
  fusion, enabling label-efficient and parameter-efficient adaptation toward
  various sensor modalities. It addresses three main challenges: 1) adaptation
  toward diverse non-RGB sensors for single-modal processing, 2) synergistic
  processing of multi-modal data via sensor fusion, and 3) mask-free training for
  different downstream tasks. Extensive experiments show that MM-SAM consistently
  outperforms SAM by large margins, demonstrating its effectiveness and
  robustness across various sensors and data modalities.
paper_authors._id: 66c41719c55655c7158d05ac
paper_authors.name: Aoran Xiao
paper_authors._id: 66c41719c55655c7158d05ad
paper_authors.name: Weihao Xuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-27T07:44:15.423Z
paper_authors._id: 65b8909c89eb3dfbe8d26780
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65b8909c89eb3dfbe8d26780/aJExXsCN9oQOdd9bsQOEO.jpeg
paper_authors.fullname: Weihao XUAN
paper_authors.type: user
paper_authors.user: weihao1115
paper_authors._id: 66c41719c55655c7158d05ae
paper_authors.name: Heli Qi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-24T07:16:13.148Z
paper_authors._id: 6682a6347ad6fe37b1c5a1d7
paper_authors.avatarUrl: /avatars/3dce84bddb82a8647580a5006c0600c0.svg
paper_authors.fullname: Heli Qi
paper_authors.type: user
paper_authors.user: qiheli1998
paper_authors._id: 66c41719c55655c7158d05af
paper_authors.name: Yun Xing
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T08:03:30.520Z
paper_authors._id: 65386066bf790526ef358701
paper_authors.avatarUrl: /avatars/f02b48f1b05c353e5ab371de45b9cf4c.svg
paper_authors.fullname: Xing Yun
paper_authors.type: user
paper_authors.user: xing0047
paper_authors._id: 66c41719c55655c7158d05b0
paper_authors.name: Naoto Yokoya
paper_authors._id: 66c41719c55655c7158d05b1
paper_authors.name: Shijian Lu
paper_title: Segment Anything with Multiple Modalities
paper_summary: Robust and accurate segmentation of scenes has become one core functionality
  in various visual recognition and navigation tasks. This has inspired the
  recent development of Segment Anything Model (SAM), a foundation model for
  general mask segmentation. However, SAM is largely tailored for single-modal
  RGB images, limiting its applicability to multi-modal data captured with
  widely-adopted sensor suites, such as LiDAR plus RGB, depth plus RGB, thermal
  plus RGB, etc. We develop MM-SAM, an extension and expansion of SAM that
  supports cross-modal and multi-modal processing for robust and enhanced
  segmentation with different sensor suites. MM-SAM features two key designs,
  namely, unsupervised cross-modal transfer and weakly-supervised multi-modal
  fusion, enabling label-efficient and parameter-efficient adaptation toward
  various sensor modalities. It addresses three main challenges: 1) adaptation
  toward diverse non-RGB sensors for single-modal processing, 2) synergistic
  processing of multi-modal data via sensor fusion, and 3) mask-free training for
  different downstream tasks. Extensive experiments show that MM-SAM consistently
  outperforms SAM by large margins, demonstrating its effectiveness and
  robustness across various sensors and data modalities.
paper_ai_keywords: Segment Anything Model (SAM) general mask segmentation single-modal RGB images LiDAR plus RGB depth plus RGB thermal plus RGB MM-SAM cross-modal and multi-modal processing unsupervised cross-modal transfer weakly-supervised multi-modal fusion label-efficient parameter-efficient single-modal processing sensor fusion mask-free training downstream tasks cross-modal multi-modal non-RGB sensors