title: Grounded 3D-LLM with Referent Tokens
summary: Prior studies on 3D scene understanding have primarily developed specialized
  models for specific tasks or required task-specific fine-tuning. In this study,
  we propose Grounded 3D-LLM, which explores the potential of 3D large
  multi-modal models (3D LMMs) to consolidate various 3D vision tasks within a
  unified generative framework. The model uses scene referent tokens as special
  noun phrases to reference 3D scenes, enabling the handling of sequences that
  interleave 3D and textual data. It offers a natural approach for translating 3D
  vision tasks into language formats using task-specific instruction templates.
  To facilitate the use of referent tokens in subsequent language modeling, we
  have curated large-scale grounded language datasets that offer finer scene-text
  correspondence at the phrase level by bootstrapping existing object labels.
  Subsequently, we introduced Contrastive LAnguage-Scene Pre-training (CLASP) to
  effectively leverage this data, thereby integrating 3D vision with language
  models. Our comprehensive evaluation covers open-ended tasks like dense
  captioning and 3D QA, alongside close-ended tasks such as object detection and
  language grounding. Experiments across multiple 3D benchmarks reveal the
  leading performance and the broad applicability of Grounded 3D-LLM. Code and
  datasets will be released on the project page:
  https://groundedscenellm.github.io/grounded_3d-llm.github.io.
paper_authors._id: 664b66b7ef47863b72c57258
paper_authors.name: Yilun Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-24T15:04:24.576Z
paper_authors._id: 642a9380ecec03b446462e51
paper_authors.avatarUrl: /avatars/5e893bf493ad66205e11aac3732b091b.svg
paper_authors.fullname: chenyilun95
paper_authors.type: user
paper_authors.user: chenyilun95
paper_authors._id: 664b66b7ef47863b72c57259
paper_authors.name: Shuai Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-21T10:24:35.136Z
paper_authors._id: 64548f6c363bb3aaf9cba136
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64548f6c363bb3aaf9cba136/HqJL9HQ5CWVOJCsHyuMOm.jpeg
paper_authors.fullname: Shuai Yang
paper_authors.type: user
paper_authors.user: ShuaiYang03
paper_authors._id: 664b66b7ef47863b72c5725a
paper_authors.name: Haifeng Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:35:02.472Z
paper_authors._id: 64d880e2bab152b247103a28
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64d880e2bab152b247103a28/2OQV-TO-kGi2XurcdBH5h.jpeg
paper_authors.fullname: Haifeng Huang
paper_authors.type: user
paper_authors.user: ZzZZCHS
paper_authors._id: 664b66b7ef47863b72c5725b
paper_authors.name: Tai Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:35:24.499Z
paper_authors._id: 64e6d9d229a548f66aff6e5b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64e6d9d229a548f66aff6e5b/yQ9E2TyzM4CfSjMPigcey.jpeg
paper_authors.fullname: Tai Wang
paper_authors.type: user
paper_authors.user: taiwang
paper_authors._id: 664b66b7ef47863b72c5725c
paper_authors.name: Ruiyuan Lyu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:35:30.884Z
paper_authors._id: 64d4ada408abd083f2cdd4bb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64d4ada408abd083f2cdd4bb/LlLDm-vk-_FGEc-KWFobw.jpeg
paper_authors.fullname: Ruiyuan Lyu
paper_authors.type: user
paper_authors.user: RuiyuanLyu
paper_authors._id: 664b66b7ef47863b72c5725d
paper_authors.name: Runsen Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:35:37.407Z
paper_authors._id: 6458b1103b81018d6b93defb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6458b1103b81018d6b93defb/JFr5j-3UO0xxlEdUksikU.jpeg
paper_authors.fullname: Runsen Xu
paper_authors.type: user
paper_authors.user: RunsenXu
paper_authors._id: 664b66b7ef47863b72c5725e
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:35:49.563Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_authors._id: 664b66b7ef47863b72c5725f
paper_authors.name: Jiangmiao Pang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-21T09:35:55.304Z
paper_authors._id: 65783ee6ee33d547aecc3ffc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65783ee6ee33d547aecc3ffc/lWZX88c-0dCsN-yB9Jhlf.jpeg
paper_authors.fullname: Jiangmiao Pang
paper_authors.type: user
paper_authors.user: Jiangmiao
paper_title: Grounded 3D-LLM with Referent Tokens
paper_summary: Prior studies on 3D scene understanding have primarily developed specialized
  models for specific tasks or required task-specific fine-tuning. In this study,
  we propose Grounded 3D-LLM, which explores the potential of 3D large
  multi-modal models (3D LMMs) to consolidate various 3D vision tasks within a
  unified generative framework. The model uses scene referent tokens as special
  noun phrases to reference 3D scenes, enabling the handling of sequences that
  interleave 3D and textual data. It offers a natural approach for translating 3D
  vision tasks into language formats using task-specific instruction templates.
  To facilitate the use of referent tokens in subsequent language modeling, we
  have curated large-scale grounded language datasets that offer finer scene-text
  correspondence at the phrase level by bootstrapping existing object labels.
  Subsequently, we introduced Contrastive LAnguage-Scene Pre-training (CLASP) to
  effectively leverage this data, thereby integrating 3D vision with language
  models. Our comprehensive evaluation covers open-ended tasks like dense
  captioning and 3D QA, alongside close-ended tasks such as object detection and
  language grounding. Experiments across multiple 3D benchmarks reveal the
  leading performance and the broad applicability of Grounded 3D-LLM. Code and
  datasets will be released on the project page:
  https://groundedscenellm.github.io/grounded_3d-llm.github.io.
paper_ai_summary: Grounded 3D-LLM integrates 3D vision tasks into a unified generative framework using 3D large multi-modal models and scene referent tokens, demonstrating superior performance across various 3D benchmarks.
paper_ai_keywords: 3D LMMs grounded 3D-LLM scene referent tokens CLASP dense captioning 3D QA object detection language grounding