title: ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and
    Planning
summary: For robots to perform a wide variety of tasks, they require a 3D
  representation of the world that is semantically rich, yet compact and
  efficient for task-driven perception and planning. Recent approaches have
  attempted to leverage features from large vision-language models to encode
  semantics in 3D representations. However, these approaches tend to produce maps
  with per-point feature vectors, which do not scale well in larger environments,
  nor do they contain semantic spatial relationships between entities in the
  environment, which are useful for downstream planning. In this work, we propose
  ConceptGraphs, an open-vocabulary graph-structured representation for 3D
  scenes. ConceptGraphs is built by leveraging 2D foundation models and fusing
  their output to 3D by multi-view association. The resulting representations
  generalize to novel semantic classes, without the need to collect large 3D
  datasets or finetune models. We demonstrate the utility of this representation
  through a number of downstream planning tasks that are specified through
  abstract (language) prompts and require complex reasoning over spatial and
  semantic concepts. (Project page: https://concept-graphs.github.io/ Explainer
  video: https://youtu.be/mRhNkQwRYnc )
paper_authors._id: 65163167217abe5d7987dc56
paper_authors.name: Qiao Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-22T12:50:16.112Z
paper_authors._id: 63d1df92f7f31a66a2d7292c
paper_authors.avatarUrl: /avatars/eb3339c1f2c82742b518b8a7f142e99a.svg
paper_authors.fullname: Qiao Gu
paper_authors.type: user
paper_authors.user: guqiao
paper_authors._id: 65163167217abe5d7987dc57
paper_authors.name: Alihusein Kuwajerwala
paper_authors._id: 65163167217abe5d7987dc58
paper_authors.name: Sacha Morin
paper_authors._id: 65163167217abe5d7987dc59
paper_authors.name: Krishna Murthy Jatavallabhula
paper_authors._id: 65163167217abe5d7987dc5a
paper_authors.name: Bipasha Sen
paper_authors._id: 65163167217abe5d7987dc5b
paper_authors.name: Aditya Agarwal
paper_authors._id: 65163167217abe5d7987dc5c
paper_authors.name: Corban Rivera
paper_authors._id: 65163167217abe5d7987dc5d
paper_authors.name: William Paul
paper_authors._id: 65163167217abe5d7987dc5e
paper_authors.name: Kirsty Ellis
paper_authors._id: 65163167217abe5d7987dc5f
paper_authors.name: Rama Chellappa
paper_authors._id: 65163167217abe5d7987dc60
paper_authors.name: Chuang Gan
paper_authors._id: 65163167217abe5d7987dc61
paper_authors.name: Celso Miguel de Melo
paper_authors._id: 65163167217abe5d7987dc62
paper_authors.name: Joshua B. Tenenbaum
paper_authors._id: 65163167217abe5d7987dc63
paper_authors.name: Antonio Torralba
paper_authors._id: 65163167217abe5d7987dc64
paper_authors.name: Florian Shkurti
paper_authors._id: 65163167217abe5d7987dc65
paper_authors.name: Liam Paull
paper_title: ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and
    Planning
paper_summary: For robots to perform a wide variety of tasks, they require a 3D
  representation of the world that is semantically rich, yet compact and
  efficient for task-driven perception and planning. Recent approaches have
  attempted to leverage features from large vision-language models to encode
  semantics in 3D representations. However, these approaches tend to produce maps
  with per-point feature vectors, which do not scale well in larger environments,
  nor do they contain semantic spatial relationships between entities in the
  environment, which are useful for downstream planning. In this work, we propose
  ConceptGraphs, an open-vocabulary graph-structured representation for 3D
  scenes. ConceptGraphs is built by leveraging 2D foundation models and fusing
  their output to 3D by multi-view association. The resulting representations
  generalize to novel semantic classes, without the need to collect large 3D
  datasets or finetune models. We demonstrate the utility of this representation
  through a number of downstream planning tasks that are specified through
  abstract (language) prompts and require complex reasoning over spatial and
  semantic concepts. (Project page: https://concept-graphs.github.io/ Explainer
  video: https://youtu.be/mRhNkQwRYnc )
paper_ai_keywords: ConceptGraphs graph-structured representation 3D scenes 2D foundation models multi-view association novel semantic classes abstract (language) prompts complex reasoning over spatial and semantic concepts