LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language
  Model as an Agent
3D visual grounding is a critical skill for household robots, enabling them
to navigate, manipulate objects, and answer questions based on their
environment. While existing approaches often rely on extensive labeled data or
exhibit limitations in handling complex language queries, we propose
LLM-Grounder, a novel zero-shot, open-vocabulary, Large Language Model
(LLM)-based 3D visual grounding pipeline. LLM-Grounder utilizes an LLM to
decompose complex natural language queries into semantic constituents and
employs a visual grounding tool, such as OpenScene or LERF, to identify objects
in a 3D scene. The LLM then evaluates the spatial and commonsense relations
among the proposed objects to make a final grounding decision. Our method does
not require any labeled training data and can generalize to novel 3D scenes and
arbitrary text queries. We evaluate LLM-Grounder on the ScanRefer benchmark and
demonstrate state-of-the-art zero-shot grounding accuracy. Our findings
indicate that LLMs significantly improve the grounding capability, especially
for complex language queries, making LLM-Grounder an effective approach for 3D
vision-language tasks in robotics. Videos and interactive demos can be found on
the project website https://chat-with-nerf.github.io/ .
650d12794d77a095225bd886
Jianing Yang
extracted_confirmed
2024-06-13T03:27:57.838Z
646afdd535c7a57f936d3ff5
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/LsDQPftQrnp_Gvanzim4J.jpeg
Jed Yang
user
jedyang97
650d12794d77a095225bd887
Xuweiyi Chen
claimed_verified
2024-09-13T07:21:20.083Z
634632aaac1cb29fb2ac9f14
https://cdn-avatars.huggingface.co/v1/production/uploads/634632aaac1cb29fb2ac9f14/nGZ2TzKOOcKMAR_NFYKkR.jpeg
Xuweiyi Chen
user
Xuweiyi
650d12794d77a095225bd888
Shengyi Qian
claimed_verified
2023-09-22T10:17:43.149Z
6445dc7e219dc6142a6a6861
https://cdn-avatars.huggingface.co/v1/production/uploads/6445dc7e219dc6142a6a6861/Ak_YV8V1NZpkF5AQnJVlJ.png
Shengyi Qian
user
shengyi-qian
650d12794d77a095225bd889
Nikhil Madaan
admin_assigned
2023-09-22T15:11:37.278Z
646be576942fd7c135dde4f7
https://cdn-avatars.huggingface.co/v1/production/uploads/646be576942fd7c135dde4f7/89f6Vgx_AlaiHM0MjpUdY.jpeg
Nikhil Madaan
user
nmadaan
650d12794d77a095225bd88a
Madhavan Iyengar
claimed_verified
2023-10-16T07:25:11.177Z
6529b8c1789d84df6fe88bec
/avatars/7a435c9ab2cf5d003d1a183229c19b6f.svg
Madhavan Iyengar
user
miyen
650d12794d77a095225bd88b
David F. Fouhey
650d12794d77a095225bd88c
Joyce Chai
LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language
  Model as an Agent
3D visual grounding is a critical skill for household robots, enabling them
to navigate, manipulate objects, and answer questions based on their
environment. While existing approaches often rely on extensive labeled data or
exhibit limitations in handling complex language queries, we propose
LLM-Grounder, a novel zero-shot, open-vocabulary, Large Language Model
(LLM)-based 3D visual grounding pipeline. LLM-Grounder utilizes an LLM to
decompose complex natural language queries into semantic constituents and
employs a visual grounding tool, such as OpenScene or LERF, to identify objects
in a 3D scene. The LLM then evaluates the spatial and commonsense relations
among the proposed objects to make a final grounding decision. Our method does
not require any labeled training data and can generalize to novel 3D scenes and
arbitrary text queries. We evaluate LLM-Grounder on the ScanRefer benchmark and
demonstrate state-of-the-art zero-shot grounding accuracy. Our findings
indicate that LLMs significantly improve the grounding capability, especially
for complex language queries, making LLM-Grounder an effective approach for 3D
vision-language tasks in robotics. Videos and interactive demos can be found on
the project website https://chat-with-nerf.github.io/ .
LLM-Grounder, a zero-shot, open-vocabulary pipeline using Large Language Models, enhances 3D visual grounding by decomposing complex queries and evaluating object relations in 3D scenes, achieving state-of-the-art accuracy on ScanRefer.
Large Language Model zero-shot open-vocabulary 3D visual grounding natural language queries semantic constituents visual grounding tool OpenScene LERF spatial relations commonsense relations grounding decision ScanRefer zero-shot grounding