title: 3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D
    Scene Understanding
summary: A 3D scene graph represents a compact scene model, storing information about
  the objects and the semantic relationships between them, making its use
  promising for robotic tasks. When interacting with a user, an embodied
  intelligent agent should be capable of responding to various queries about the
  scene formulated in natural language. Large Language Models (LLMs) are
  beneficial solutions for user-robot interaction due to their natural language
  understanding and reasoning abilities. Recent methods for creating learnable
  representations of 3D scenes have demonstrated the potential to improve the
  quality of LLMs responses by adapting to the 3D world. However, the existing
  methods do not explicitly utilize information about the semantic relationships
  between objects, limiting themselves to information about their coordinates. In
  this work, we propose a method 3DGraphLLM for constructing a learnable
  representation of a 3D scene graph. The learnable representation is used as
  input for LLMs to perform 3D vision-language tasks. In our experiments on
  popular ScanRefer, RIORefer, Multi3DRefer, ScanQA, Sqa3D, and Scan2cap
  datasets, we demonstrate the advantage of this approach over baseline methods
  that do not use information about the semantic relationships between objects.
  The code is publicly available at
  https://github.com/CognitiveAISystems/3DGraphLLM.
paper_authors._id: 676bbe579484d105b89dba3b
paper_authors.name: Tatiana Zemskova
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:32:37.677Z
paper_authors._id: 6363767e572fd34304f49a67
paper_authors.avatarUrl: /avatars/a9fc92b6005d48adf71a45bebf812648.svg
paper_authors.fullname: Tatiana Zemskova
paper_authors.type: user
paper_authors.user: wingrune
paper_authors._id: 676bbe579484d105b89dba3c
paper_authors.name: Dmitry Yudin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:32:35.428Z
paper_authors._id: 67585775ab7e701b5754649f
paper_authors.avatarUrl: /avatars/7ccaf915b6b409e49d35f5212b8293c3.svg
paper_authors.fullname: Dmitry Yudin
paper_authors.type: user
paper_authors.user: yuddim
paper_title: 3DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D
    Scene Understanding
paper_summary: A 3D scene graph represents a compact scene model, storing information about
  the objects and the semantic relationships between them, making its use
  promising for robotic tasks. When interacting with a user, an embodied
  intelligent agent should be capable of responding to various queries about the
  scene formulated in natural language. Large Language Models (LLMs) are
  beneficial solutions for user-robot interaction due to their natural language
  understanding and reasoning abilities. Recent methods for creating learnable
  representations of 3D scenes have demonstrated the potential to improve the
  quality of LLMs responses by adapting to the 3D world. However, the existing
  methods do not explicitly utilize information about the semantic relationships
  between objects, limiting themselves to information about their coordinates. In
  this work, we propose a method 3DGraphLLM for constructing a learnable
  representation of a 3D scene graph. The learnable representation is used as
  input for LLMs to perform 3D vision-language tasks. In our experiments on
  popular ScanRefer, RIORefer, Multi3DRefer, ScanQA, Sqa3D, and Scan2cap
  datasets, we demonstrate the advantage of this approach over baseline methods
  that do not use information about the semantic relationships between objects.
  The code is publicly available at
  https://github.com/CognitiveAISystems/3DGraphLLM.
paper_ai_summary: A 3D scene graph representation integrated with large language models enhances user-robot interaction by incorporating semantic relationships between objects, leading to improved performance in 3D vision-language tasks compared to baselines.
paper_ai_keywords: 3D scene graph learnable representation large language models semantic relationships coordinates ScanRefer RIORefer Multi3DRefer ScanQA Sqa3D Scan2cap 3D vision-language tasks