title: M3: 3D-Spatial MultiModal Memory
summary: We present 3D Spatial MultiModal Memory (M3), a multimodal memory system
  designed to retain information about medium-sized static scenes through video
  sources for visual perception. By integrating 3D Gaussian Splatting techniques
  with foundation models, M3 builds a multimodal memory capable of rendering
  feature representations across granularities, encompassing a wide range of
  knowledge. In our exploration, we identify two key challenges in previous works
  on feature splatting: (1) computational constraints in storing high-dimensional
  features for each Gaussian primitive, and (2) misalignment or information loss
  between distilled features and foundation model features. To address these
  challenges, we propose M3 with key components of principal scene components and
  Gaussian memory attention, enabling efficient training and inference. To
  validate M3, we conduct comprehensive quantitative evaluations of feature
  similarity and downstream tasks, as well as qualitative visualizations to
  highlight the pixel trace of Gaussian memory attention. Our approach
  encompasses a diverse range of foundation models, including vision-language
  models (VLMs), perception models, and large multimodal and language models
  (LMMs/LLMs). Furthermore, to demonstrate real-world applicability, we deploy
  M3's feature field in indoor scenes on a quadruped robot. Notably, we claim
  that M3 is the first work to address the core compression challenges in 3D
  feature distillation.
paper_authors._id: 67dccb8ca33f11a56567bd61
paper_authors.name: Xueyan Zou
paper_authors._id: 67dccb8ca33f11a56567bd62
paper_authors.name: Yuchen Song
paper_authors._id: 67dccb8ca33f11a56567bd63
paper_authors.name: Ri-Zhao Qiu
paper_authors._id: 67dccb8ca33f11a56567bd64
paper_authors.name: Xuanbin Peng
paper_authors._id: 67dccb8ca33f11a56567bd65
paper_authors.name: Jianglong Ye
paper_authors._id: 67dccb8ca33f11a56567bd66
paper_authors.name: Sifei Liu
paper_authors._id: 67dccb8ca33f11a56567bd67
paper_authors.name: Xiaolong Wang
paper_title: M3: 3D-Spatial MultiModal Memory
paper_summary: We present 3D Spatial MultiModal Memory (M3), a multimodal memory system
  designed to retain information about medium-sized static scenes through video
  sources for visual perception. By integrating 3D Gaussian Splatting techniques
  with foundation models, M3 builds a multimodal memory capable of rendering
  feature representations across granularities, encompassing a wide range of
  knowledge. In our exploration, we identify two key challenges in previous works
  on feature splatting: (1) computational constraints in storing high-dimensional
  features for each Gaussian primitive, and (2) misalignment or information loss
  between distilled features and foundation model features. To address these
  challenges, we propose M3 with key components of principal scene components and
  Gaussian memory attention, enabling efficient training and inference. To
  validate M3, we conduct comprehensive quantitative evaluations of feature
  similarity and downstream tasks, as well as qualitative visualizations to
  highlight the pixel trace of Gaussian memory attention. Our approach
  encompasses a diverse range of foundation models, including vision-language
  models (VLMs), perception models, and large multimodal and language models
  (LMMs/LLMs). Furthermore, to demonstrate real-world applicability, we deploy
  M3's feature field in indoor scenes on a quadruped robot. Notably, we claim
  that M3 is the first work to address the core compression challenges in 3D
  feature distillation.
paper_ai_summary: M3, a multimodal memory system combining 3D Gaussian Splatting and foundation models, addresses compression and alignment issues for high-dimensional scene features, enhancing efficiency and real-world applicability.
paper_ai_keywords: 3D Spatial MultiModal Memory M3 3D Gaussian Splatting multimodal memory feature representations Gaussian primitives principal scene components Gaussian memory attention vision-language models perception models large multimodal models large language models