title: 3D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and
    Less Hallucination
summary: The integration of language and 3D perception is crucial for developing
  embodied agents and robots that comprehend and interact with the physical
  world. While large language models (LLMs) have demonstrated impressive language
  understanding and generation capabilities, their adaptation to 3D environments
  (3D-LLMs) remains in its early stages. A primary challenge is the absence of
  large-scale datasets that provide dense grounding between language and 3D
  scenes. In this paper, we introduce 3D-GRAND, a pioneering large-scale dataset
  comprising 40,087 household scenes paired with 6.2 million densely-grounded
  scene-language instructions. Our results show that instruction tuning with
  3D-GRAND significantly enhances grounding capabilities and reduces
  hallucinations in 3D-LLMs. As part of our contributions, we propose a
  comprehensive benchmark 3D-POPE to systematically evaluate hallucination in
  3D-LLMs, enabling fair comparisons among future models. Our experiments
  highlight a scaling effect between dataset size and 3D-LLM performance,
  emphasizing the critical role of large-scale 3D-text datasets in advancing
  embodied AI research. Notably, our results demonstrate early signals for
  effective sim-to-real transfer, indicating that models trained on large
  synthetic data can perform well on real-world 3D scans. Through 3D-GRAND and
  3D-POPE, we aim to equip the embodied AI community with essential resources and
  insights, setting the stage for more reliable and better-grounded 3D-LLMs.
  Project website: https://3d-grand.github.io
paper_authors._id: 666a643200bacf0820a4c016
paper_authors.name: Jianing Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-13T09:31:01.780Z
paper_authors._id: 646afdd535c7a57f936d3ff5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/LsDQPftQrnp_Gvanzim4J.jpeg
paper_authors.fullname: Jed Yang
paper_authors.type: user
paper_authors.user: jedyang97
paper_authors._id: 666a643200bacf0820a4c017
paper_authors.name: Xuweiyi Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T20:01:02.826Z
paper_authors._id: 634632aaac1cb29fb2ac9f14
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/634632aaac1cb29fb2ac9f14/nGZ2TzKOOcKMAR_NFYKkR.jpeg
paper_authors.fullname: Xuweiyi Chen
paper_authors.type: user
paper_authors.user: Xuweiyi
paper_authors._id: 666a643200bacf0820a4c018
paper_authors.name: Nikhil Madaan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T20:01:09.150Z
paper_authors._id: 646be576942fd7c135dde4f7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646be576942fd7c135dde4f7/89f6Vgx_AlaiHM0MjpUdY.jpeg
paper_authors.fullname: Nikhil Madaan
paper_authors.type: user
paper_authors.user: nmadaan
paper_authors._id: 666a643200bacf0820a4c019
paper_authors.name: Madhavan Iyengar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T20:01:14.622Z
paper_authors._id: 6529b8c1789d84df6fe88bec
paper_authors.avatarUrl: /avatars/7a435c9ab2cf5d003d1a183229c19b6f.svg
paper_authors.fullname: Madhavan Iyengar
paper_authors.type: user
paper_authors.user: miyen
paper_authors._id: 666a643200bacf0820a4c01a
paper_authors.name: Shengyi Qian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T20:01:20.041Z
paper_authors._id: 6445dc7e219dc6142a6a6861
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6445dc7e219dc6142a6a6861/Ak_YV8V1NZpkF5AQnJVlJ.png
paper_authors.fullname: Shengyi Qian
paper_authors.type: user
paper_authors.user: shengyi-qian
paper_authors._id: 666a643200bacf0820a4c01b
paper_authors.name: David F. Fouhey
paper_authors._id: 666a643200bacf0820a4c01c
paper_authors.name: Joyce Chai
paper_title: 3D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and
    Less Hallucination
paper_summary: The integration of language and 3D perception is crucial for developing
  embodied agents and robots that comprehend and interact with the physical
  world. While large language models (LLMs) have demonstrated impressive language
  understanding and generation capabilities, their adaptation to 3D environments
  (3D-LLMs) remains in its early stages. A primary challenge is the absence of
  large-scale datasets that provide dense grounding between language and 3D
  scenes. In this paper, we introduce 3D-GRAND, a pioneering large-scale dataset
  comprising 40,087 household scenes paired with 6.2 million densely-grounded
  scene-language instructions. Our results show that instruction tuning with
  3D-GRAND significantly enhances grounding capabilities and reduces
  hallucinations in 3D-LLMs. As part of our contributions, we propose a
  comprehensive benchmark 3D-POPE to systematically evaluate hallucination in
  3D-LLMs, enabling fair comparisons among future models. Our experiments
  highlight a scaling effect between dataset size and 3D-LLM performance,
  emphasizing the critical role of large-scale 3D-text datasets in advancing
  embodied AI research. Notably, our results demonstrate early signals for
  effective sim-to-real transfer, indicating that models trained on large
  synthetic data can perform well on real-world 3D scans. Through 3D-GRAND and
  3D-POPE, we aim to equip the embodied AI community with essential resources and
  insights, setting the stage for more reliable and better-grounded 3D-LLMs.
  Project website: https://3d-grand.github.io
paper_ai_summary: 3D-GRAND, a large-scale dataset, enhances 3D-LLM performance and hallucination reduction through instruction tuning and establishes a benchmark for sim-to-real transfer.
paper_ai_keywords: large language models 3D-LLMs 3D-GRAND 3D-POPE scene-language instructions grounding capabilities hallucination benchmark sim-to-real transfer