title: FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D
    Scene Understanding
summary: Precisely perceiving the geometric and semantic properties of real-world 3D
  objects is crucial for the continued evolution of augmented reality and robotic
  applications. To this end, we present  (), which
  incorporates vision-language embeddings of foundation models into 3D Gaussian
  Splatting (GS). The key contribution of this work is an efficient method to
  reconstruct and represent 3D vision-language models. This is achieved by
  distilling feature maps generated from image-based foundation models into those
  rendered from our 3D model. To ensure high-quality rendering and fast training,
  we introduce a novel scene representation by integrating strengths from both GS
  and multi-resolution hash encodings (MHE). Our effective training procedure
  also introduces a pixel alignment loss that makes the rendered feature distance
  of same semantic entities close, following the pixel-level semantic boundaries.
  Our results demonstrate remarkable multi-view semantic consistency,
  facilitating diverse downstream tasks, beating state-of-the-art methods by
  10.2 percent on open-vocabulary language-based object detection,
  despite that we are 851times faster for inference. This research
  explores the intersection of vision, language, and 3D scene representation,
  paving the way for enhanced scene understanding in uncontrolled real-world
  environments. We plan to release the code upon paper acceptance.
paper_authors._id: 6597801f9d1469e51560ef5e
paper_authors.name: Xingxing Zuo
paper_authors._id: 6597801f9d1469e51560ef5f
paper_authors.name: Pouya Samangouei
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-04-22T17:14:17.178Z
paper_authors._id: 65e799061649b1896fdfc80d
paper_authors.avatarUrl: /avatars/56db739ff7458138b6f708132292a73f.svg
paper_authors.fullname: Pouya Samangouei
paper_authors.type: user
paper_authors.user: samangoueigoogle
paper_authors._id: 6597801f9d1469e51560ef60
paper_authors.name: Yunwen Zhou
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-01-03T08:16:42.769Z
paper_authors._id: 67779cc2202208d2df41b4c3
paper_authors.avatarUrl: /avatars/19744872134d0a12a667d306c1d23009.svg
paper_authors.fullname: Verse Zhou
paper_authors.type: user
paper_authors.user: versegg1994
paper_authors._id: 6597801f9d1469e51560ef61
paper_authors.name: Yan Di
paper_authors._id: 6597801f9d1469e51560ef62
paper_authors.name: Mingyang Li
paper_title: FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D
    Scene Understanding
paper_summary: Precisely perceiving the geometric and semantic properties of real-world 3D
  objects is crucial for the continued evolution of augmented reality and robotic
  applications. To this end, we present  (), which
  incorporates vision-language embeddings of foundation models into 3D Gaussian
  Splatting (GS). The key contribution of this work is an efficient method to
  reconstruct and represent 3D vision-language models. This is achieved by
  distilling feature maps generated from image-based foundation models into those
  rendered from our 3D model. To ensure high-quality rendering and fast training,
  we introduce a novel scene representation by integrating strengths from both GS
  and multi-resolution hash encodings (MHE). Our effective training procedure
  also introduces a pixel alignment loss that makes the rendered feature distance
  of same semantic entities close, following the pixel-level semantic boundaries.
  Our results demonstrate remarkable multi-view semantic consistency,
  facilitating diverse downstream tasks, beating state-of-the-art methods by
  10.2 percent on open-vocabulary language-based object detection,
  despite that we are 851times faster for inference. This research
  explores the intersection of vision, language, and 3D scene representation,
  paving the way for enhanced scene understanding in uncontrolled real-world
  environments. We plan to release the code upon paper acceptance.
paper_ai_summary: The work introduces a method to integrate vision-language embeddings into 3D Gaussian Splatting for efficient 3D reconstruction, improving multi-view semantic consistency and inference speed.
paper_ai_keywords: Gaussian Splatting multi-resolution hash encodings pixel alignment loss scene representation foundation models feature distillation