title: LangSplat: 3D Language Gaussian Splatting
summary: Human lives in a 3D world and commonly uses natural language to interact with
  a 3D scene. Modeling a 3D language field to support open-ended language queries
  in 3D has gained increasing attention recently. This paper introduces
  LangSplat, which constructs a 3D language field that enables precise and
  efficient open-vocabulary querying within 3D spaces. Unlike existing methods
  that ground CLIP language embeddings in a NeRF model, LangSplat advances the
  field by utilizing a collection of 3D Gaussians, each encoding language
  features distilled from CLIP, to represent the language field. By employing a
  tile-based splatting technique for rendering language features, we circumvent
  the costly rendering process inherent in NeRF. Instead of directly learning
  CLIP embeddings, LangSplat first trains a scene-wise language autoencoder and
  then learns language features on the scene-specific latent space, thereby
  alleviating substantial memory demands imposed by explicit modeling. Existing
  methods struggle with imprecise and vague 3D language fields, which fail to
  discern clear boundaries between objects. We delve into this issue and propose
  to learn hierarchical semantics using SAM, thereby eliminating the need for
  extensively querying the language field across various scales and the
  regularization of DINO features. Extensive experiments on open-vocabulary 3D
  object localization and semantic segmentation demonstrate that LangSplat
  significantly outperforms the previous state-of-the-art method LERF by a large
  margin. Notably, LangSplat is extremely efficient, achieving a {\speed}
  times speedup compared to LERF at the resolution of 1440 times 1080. We
  strongly recommend readers to check out our video results at
  https://langsplat.github.io
paper_authors._id: 658bba53ddfe71efe7db540f
paper_authors.name: Minghan Qin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-29T09:00:41.663Z
paper_authors._id: 642002b51ccd411979d72b18
paper_authors.avatarUrl: /avatars/0a1e7b632fb30ad39e363c3838117322.svg
paper_authors.fullname: Minghan Qin
paper_authors.type: user
paper_authors.user: Qmh
paper_authors._id: 658bba53ddfe71efe7db5410
paper_authors.name: Wanhua Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-27T11:05:46.069Z
paper_authors._id: 658bb7e47459b6e471b9d2e6
paper_authors.avatarUrl: /avatars/efd8051b468b4dbcb5d149479de67c58.svg
paper_authors.fullname: Wanhua Li
paper_authors.type: user
paper_authors.user: EthanTaylor
paper_authors._id: 658bba53ddfe71efe7db5411
paper_authors.name: Jiawei Zhou
paper_authors._id: 658bba53ddfe71efe7db5412
paper_authors.name: Haoqian Wang
paper_authors._id: 658bba53ddfe71efe7db5413
paper_authors.name: Hanspeter Pfister
paper_title: LangSplat: 3D Language Gaussian Splatting
paper_summary: Human lives in a 3D world and commonly uses natural language to interact with
  a 3D scene. Modeling a 3D language field to support open-ended language queries
  in 3D has gained increasing attention recently. This paper introduces
  LangSplat, which constructs a 3D language field that enables precise and
  efficient open-vocabulary querying within 3D spaces. Unlike existing methods
  that ground CLIP language embeddings in a NeRF model, LangSplat advances the
  field by utilizing a collection of 3D Gaussians, each encoding language
  features distilled from CLIP, to represent the language field. By employing a
  tile-based splatting technique for rendering language features, we circumvent
  the costly rendering process inherent in NeRF. Instead of directly learning
  CLIP embeddings, LangSplat first trains a scene-wise language autoencoder and
  then learns language features on the scene-specific latent space, thereby
  alleviating substantial memory demands imposed by explicit modeling. Existing
  methods struggle with imprecise and vague 3D language fields, which fail to
  discern clear boundaries between objects. We delve into this issue and propose
  to learn hierarchical semantics using SAM, thereby eliminating the need for
  extensively querying the language field across various scales and the
  regularization of DINO features. Extensive experiments on open-vocabulary 3D
  object localization and semantic segmentation demonstrate that LangSplat
  significantly outperforms the previous state-of-the-art method LERF by a large
  margin. Notably, LangSplat is extremely efficient, achieving a {\speed}
  times speedup compared to LERF at the resolution of 1440 times 1080. We
  strongly recommend readers to check out our video results at
  https://langsplat.github.io
paper_ai_summary: LangSplat constructs a 3D language field using a splatting technique with hierarchical semantics, enabling precise and efficient open-vocabulary querying in 3D spaces while significantly outperforming existing methods.
paper_ai_keywords: CLIP NeRF 3D Gaussians tile-based splatting scene-wise language autoencoder SAM hierarchical semantics DINO features open-vocabulary 3D object localization semantic segmentation LERF