ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and
  Planning
For robots to perform a wide variety of tasks, they require a 3D
representation of the world that is semantically rich, yet compact and
efficient for task-driven perception and planning. Recent approaches have
attempted to leverage features from large vision-language models to encode
semantics in 3D representations. However, these approaches tend to produce maps
with per-point feature vectors, which do not scale well in larger environments,
nor do they contain semantic spatial relationships between entities in the
environment, which are useful for downstream planning. In this work, we propose
ConceptGraphs, an open-vocabulary graph-structured representation for 3D
scenes. ConceptGraphs is built by leveraging 2D foundation models and fusing
their output to 3D by multi-view association. The resulting representations
generalize to novel semantic classes, without the need to collect large 3D
datasets or finetune models. We demonstrate the utility of this representation
through a number of downstream planning tasks that are specified through
abstract (language) prompts and require complex reasoning over spatial and
semantic concepts. (Project page: https://concept-graphs.github.io/ Explainer
video: https://youtu.be/mRhNkQwRYnc )
65163167217abe5d7987dc56
Qiao Gu
claimed_verified
2024-09-22T12:50:16.112Z
63d1df92f7f31a66a2d7292c
/avatars/eb3339c1f2c82742b518b8a7f142e99a.svg
Qiao Gu
user
guqiao
65163167217abe5d7987dc57
Alihusein Kuwajerwala
65163167217abe5d7987dc58
Sacha Morin
65163167217abe5d7987dc59
Krishna Murthy Jatavallabhula
65163167217abe5d7987dc5a
Bipasha Sen
65163167217abe5d7987dc5b
Aditya Agarwal
65163167217abe5d7987dc5c
Corban Rivera
65163167217abe5d7987dc5d
William Paul
65163167217abe5d7987dc5e
Kirsty Ellis
65163167217abe5d7987dc5f
Rama Chellappa
65163167217abe5d7987dc60
Chuang Gan
65163167217abe5d7987dc61
Celso Miguel de Melo
65163167217abe5d7987dc62
Joshua B. Tenenbaum
65163167217abe5d7987dc63
Antonio Torralba
65163167217abe5d7987dc64
Florian Shkurti
65163167217abe5d7987dc65
Liam Paull
ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and
  Planning
For robots to perform a wide variety of tasks, they require a 3D
representation of the world that is semantically rich, yet compact and
efficient for task-driven perception and planning. Recent approaches have
attempted to leverage features from large vision-language models to encode
semantics in 3D representations. However, these approaches tend to produce maps
with per-point feature vectors, which do not scale well in larger environments,
nor do they contain semantic spatial relationships between entities in the
environment, which are useful for downstream planning. In this work, we propose
ConceptGraphs, an open-vocabulary graph-structured representation for 3D
scenes. ConceptGraphs is built by leveraging 2D foundation models and fusing
their output to 3D by multi-view association. The resulting representations
generalize to novel semantic classes, without the need to collect large 3D
datasets or finetune models. We demonstrate the utility of this representation
through a number of downstream planning tasks that are specified through
abstract (language) prompts and require complex reasoning over spatial and
semantic concepts. (Project page: https://concept-graphs.github.io/ Explainer
video: https://youtu.be/mRhNkQwRYnc )
ConceptGraphs, a graph-structured representation leveraging 2D foundation models, enables semantically rich and scalable 3D scene understanding for task-driven perception and planning.
ConceptGraphs graph-structured representation 3D scene 2D foundation models multi-view association open-vocabulary spatial relationships semantic classes abstract prompts complex reasoning