Grounded 3D-LLM with Referent Tokens
Prior studies on 3D scene understanding have primarily developed specialized
models for specific tasks or required task-specific fine-tuning. In this study,
we propose Grounded 3D-LLM, which explores the potential of 3D large
multi-modal models (3D LMMs) to consolidate various 3D vision tasks within a
unified generative framework. The model uses scene referent tokens as special
noun phrases to reference 3D scenes, enabling the handling of sequences that
interleave 3D and textual data. It offers a natural approach for translating 3D
vision tasks into language formats using task-specific instruction templates.
To facilitate the use of referent tokens in subsequent language modeling, we
have curated large-scale grounded language datasets that offer finer scene-text
correspondence at the phrase level by bootstrapping existing object labels.
Subsequently, we introduced Contrastive LAnguage-Scene Pre-training (CLASP) to
effectively leverage this data, thereby integrating 3D vision with language
models. Our comprehensive evaluation covers open-ended tasks like dense
captioning and 3D QA, alongside close-ended tasks such as object detection and
language grounding. Experiments across multiple 3D benchmarks reveal the
leading performance and the broad applicability of Grounded 3D-LLM. Code and
datasets will be released on the project page:
https://groundedscenellm.github.io/grounded_3d-llm.github.io.
664b66b7ef47863b72c57258
Yilun Chen
admin_assigned
2024-05-24T15:04:24.576Z
642a9380ecec03b446462e51
/avatars/5e893bf493ad66205e11aac3732b091b.svg
chenyilun95
user
chenyilun95
664b66b7ef47863b72c57259
Shuai Yang
claimed_verified
2024-06-21T10:24:35.136Z
64548f6c363bb3aaf9cba136
https://cdn-avatars.huggingface.co/v1/production/uploads/64548f6c363bb3aaf9cba136/HqJL9HQ5CWVOJCsHyuMOm.jpeg
Shuai Yang
user
ShuaiYang03
664b66b7ef47863b72c5725a
Haifeng Huang
admin_assigned
2024-05-21T09:35:02.472Z
64d880e2bab152b247103a28
https://cdn-avatars.huggingface.co/v1/production/uploads/64d880e2bab152b247103a28/2OQV-TO-kGi2XurcdBH5h.jpeg
Haifeng Huang
user
ZzZZCHS
664b66b7ef47863b72c5725b
Tai Wang
admin_assigned
2024-05-21T09:35:24.499Z
64e6d9d229a548f66aff6e5b
https://cdn-avatars.huggingface.co/v1/production/uploads/64e6d9d229a548f66aff6e5b/yQ9E2TyzM4CfSjMPigcey.jpeg
Tai Wang
user
taiwang
664b66b7ef47863b72c5725c
Ruiyuan Lyu
admin_assigned
2024-05-21T09:35:30.884Z
64d4ada408abd083f2cdd4bb
https://cdn-avatars.huggingface.co/v1/production/uploads/64d4ada408abd083f2cdd4bb/LlLDm-vk-_FGEc-KWFobw.jpeg
Ruiyuan Lyu
user
RuiyuanLyu
664b66b7ef47863b72c5725d
Runsen Xu
admin_assigned
2024-05-21T09:35:37.407Z
6458b1103b81018d6b93defb
https://cdn-avatars.huggingface.co/v1/production/uploads/6458b1103b81018d6b93defb/JFr5j-3UO0xxlEdUksikU.jpeg
Runsen Xu
user
RunsenXu
664b66b7ef47863b72c5725e
Dahua Lin
admin_assigned
2024-05-21T09:35:49.563Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
664b66b7ef47863b72c5725f
Jiangmiao Pang
admin_assigned
2024-05-21T09:35:55.304Z
65783ee6ee33d547aecc3ffc
https://cdn-avatars.huggingface.co/v1/production/uploads/65783ee6ee33d547aecc3ffc/lWZX88c-0dCsN-yB9Jhlf.jpeg
Jiangmiao Pang
user
Jiangmiao
Grounded 3D-LLM with Referent Tokens
Prior studies on 3D scene understanding have primarily developed specialized
models for specific tasks or required task-specific fine-tuning. In this study,
we propose Grounded 3D-LLM, which explores the potential of 3D large
multi-modal models (3D LMMs) to consolidate various 3D vision tasks within a
unified generative framework. The model uses scene referent tokens as special
noun phrases to reference 3D scenes, enabling the handling of sequences that
interleave 3D and textual data. It offers a natural approach for translating 3D
vision tasks into language formats using task-specific instruction templates.
To facilitate the use of referent tokens in subsequent language modeling, we
have curated large-scale grounded language datasets that offer finer scene-text
correspondence at the phrase level by bootstrapping existing object labels.
Subsequently, we introduced Contrastive LAnguage-Scene Pre-training (CLASP) to
effectively leverage this data, thereby integrating 3D vision with language
models. Our comprehensive evaluation covers open-ended tasks like dense
captioning and 3D QA, alongside close-ended tasks such as object detection and
language grounding. Experiments across multiple 3D benchmarks reveal the
leading performance and the broad applicability of Grounded 3D-LLM. Code and
datasets will be released on the project page:
https://groundedscenellm.github.io/grounded_3d-llm.github.io.
Grounded 3D-LLM integrates 3D vision tasks into a unified generative framework using 3D large multi-modal models and scene referent tokens, demonstrating superior performance across various 3D benchmarks.
3D LMMs grounded 3D-LLM scene referent tokens CLASP dense captioning 3D QA object detection language grounding