title: LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language
    Model as an Agent
summary: 3D visual grounding is a critical skill for household robots, enabling them
  to navigate, manipulate objects, and answer questions based on their
  environment. While existing approaches often rely on extensive labeled data or
  exhibit limitations in handling complex language queries, we propose
  LLM-Grounder, a novel zero-shot, open-vocabulary, Large Language Model
  (LLM)-based 3D visual grounding pipeline. LLM-Grounder utilizes an LLM to
  decompose complex natural language queries into semantic constituents and
  employs a visual grounding tool, such as OpenScene or LERF, to identify objects
  in a 3D scene. The LLM then evaluates the spatial and commonsense relations
  among the proposed objects to make a final grounding decision. Our method does
  not require any labeled training data and can generalize to novel 3D scenes and
  arbitrary text queries. We evaluate LLM-Grounder on the ScanRefer benchmark and
  demonstrate state-of-the-art zero-shot grounding accuracy. Our findings
  indicate that LLMs significantly improve the grounding capability, especially
  for complex language queries, making LLM-Grounder an effective approach for 3D
  vision-language tasks in robotics. Videos and interactive demos can be found on
  the project website https://chat-with-nerf.github.io/ .
paper_authors._id: 650d12794d77a095225bd886
paper_authors.name: Jianing Yang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-13T03:27:57.838Z
paper_authors._id: 646afdd535c7a57f936d3ff5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/LsDQPftQrnp_Gvanzim4J.jpeg
paper_authors.fullname: Jed Yang
paper_authors.type: user
paper_authors.user: jedyang97
paper_authors._id: 650d12794d77a095225bd887
paper_authors.name: Xuweiyi Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-13T07:21:20.083Z
paper_authors._id: 634632aaac1cb29fb2ac9f14
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/634632aaac1cb29fb2ac9f14/nGZ2TzKOOcKMAR_NFYKkR.jpeg
paper_authors.fullname: Xuweiyi Chen
paper_authors.type: user
paper_authors.user: Xuweiyi
paper_authors._id: 650d12794d77a095225bd888
paper_authors.name: Shengyi Qian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-22T10:17:43.149Z
paper_authors._id: 6445dc7e219dc6142a6a6861
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6445dc7e219dc6142a6a6861/Ak_YV8V1NZpkF5AQnJVlJ.png
paper_authors.fullname: Shengyi Qian
paper_authors.type: user
paper_authors.user: shengyi-qian
paper_authors._id: 650d12794d77a095225bd889
paper_authors.name: Nikhil Madaan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-22T15:11:37.278Z
paper_authors._id: 646be576942fd7c135dde4f7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646be576942fd7c135dde4f7/89f6Vgx_AlaiHM0MjpUdY.jpeg
paper_authors.fullname: Nikhil Madaan
paper_authors.type: user
paper_authors.user: nmadaan
paper_authors._id: 650d12794d77a095225bd88a
paper_authors.name: Madhavan Iyengar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-16T07:25:11.177Z
paper_authors._id: 6529b8c1789d84df6fe88bec
paper_authors.avatarUrl: /avatars/7a435c9ab2cf5d003d1a183229c19b6f.svg
paper_authors.fullname: Madhavan Iyengar
paper_authors.type: user
paper_authors.user: miyen
paper_authors._id: 650d12794d77a095225bd88b
paper_authors.name: David F. Fouhey
paper_authors._id: 650d12794d77a095225bd88c
paper_authors.name: Joyce Chai
paper_title: LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language
    Model as an Agent
paper_summary: 3D visual grounding is a critical skill for household robots, enabling them
  to navigate, manipulate objects, and answer questions based on their
  environment. While existing approaches often rely on extensive labeled data or
  exhibit limitations in handling complex language queries, we propose
  LLM-Grounder, a novel zero-shot, open-vocabulary, Large Language Model
  (LLM)-based 3D visual grounding pipeline. LLM-Grounder utilizes an LLM to
  decompose complex natural language queries into semantic constituents and
  employs a visual grounding tool, such as OpenScene or LERF, to identify objects
  in a 3D scene. The LLM then evaluates the spatial and commonsense relations
  among the proposed objects to make a final grounding decision. Our method does
  not require any labeled training data and can generalize to novel 3D scenes and
  arbitrary text queries. We evaluate LLM-Grounder on the ScanRefer benchmark and
  demonstrate state-of-the-art zero-shot grounding accuracy. Our findings
  indicate that LLMs significantly improve the grounding capability, especially
  for complex language queries, making LLM-Grounder an effective approach for 3D
  vision-language tasks in robotics. Videos and interactive demos can be found on
  the project website https://chat-with-nerf.github.io/ .
paper_ai_summary: LLM-Grounder, a zero-shot, open-vocabulary pipeline using Large Language Models, enhances 3D visual grounding by decomposing complex queries and evaluating object relations in 3D scenes, achieving state-of-the-art accuracy on ScanRefer.
paper_ai_keywords: Large Language Model zero-shot open-vocabulary 3D visual grounding natural language queries semantic constituents visual grounding tool OpenScene LERF spatial relations commonsense relations grounding decision ScanRefer zero-shot grounding