title: 3D-LLM: Injecting the 3D World into Large Language Models
summary: Large language models (LLMs) and Vision-Language Models (VLMs) have been
  proven to excel at multiple tasks, such as commonsense reasoning. Powerful as
  these models can be, they are not grounded in the 3D physical world, which
  involves richer concepts such as spatial relationships, affordances, physics,
  layout, and so on. In this work, we propose to inject the 3D world into large
  language models and introduce a whole new family of 3D-LLMs. Specifically,
  3D-LLMs can take 3D point clouds and their features as input and perform a
  diverse set of 3D-related tasks, including captioning, dense captioning, 3D
  question answering, task decomposition, 3D grounding, 3D-assisted dialog,
  navigation, and so on. Using three types of prompting mechanisms that we
  design, we are able to collect over 300k 3D-language data covering these tasks.
  To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that
  obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as
  our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism,
  3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show
  that our model outperforms state-of-the-art baselines by a large margin (e.g.,
  the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore,
  experiments on our held-in datasets for 3D captioning, task composition, and
  3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative
  examples also show that our model could perform more tasks beyond the scope of
  existing LLMs and VLMs. Project Page: : https://vis-www.cs.umass.edu/3dllm/.
paper_authors._id: 64bf5732796f20daadaa7b78
paper_authors.name: Yining Hong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-25T08:27:15.730Z
paper_authors._id: 6431b64df76c34519e93d1ba
paper_authors.avatarUrl: /avatars/ea577762b6b4798f87a7a3f1d53d082c.svg
paper_authors.fullname: Yining Hong
paper_authors.type: user
paper_authors.user: evelynhong
paper_authors._id: 64bf5732796f20daadaa7b79
paper_authors.name: Haoyu Zhen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-25T08:24:41.928Z
paper_authors._id: 6437c7dae282b4a48eaf065e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6437c7dae282b4a48eaf065e/AxodKQXyrviTFQRyjnL01.jpeg
paper_authors.fullname: Haoyu Zhen
paper_authors.type: user
paper_authors.user: anyeZHY
paper_authors._id: 64bf5732796f20daadaa7b7a
paper_authors.name: Peihao Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-26T08:43:57.546Z
paper_authors._id: 64bfe895c874dda773303b44
paper_authors.avatarUrl: /avatars/a54ddb6d69a73f63f9d848f28abcbff3.svg
paper_authors.fullname: PeihaoChen
paper_authors.type: user
paper_authors.user: PeihaoChen
paper_authors._id: 64bf5732796f20daadaa7b7b
paper_authors.name: Shuhong Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-25T08:25:54.807Z
paper_authors._id: 64361417a4bd75c62cc1e534
paper_authors.avatarUrl: /avatars/b9b43985a0190ccf418c6853f9c3eb16.svg
paper_authors.fullname: Shuhong Zheng
paper_authors.type: user
paper_authors.user: ShuhongZheng
paper_authors._id: 64bf5732796f20daadaa7b7c
paper_authors.name: Yilun Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-25T08:26:20.418Z
paper_authors._id: 63c9bd445fdc575773c732fe
paper_authors.avatarUrl: /avatars/def472d1ab3fbf751225357c0932ae7e.svg
paper_authors.fullname: Yilun Du
paper_authors.type: user
paper_authors.user: yilundu
paper_authors._id: 64bf5732796f20daadaa7b7d
paper_authors.name: Zhenfang Chen
paper_authors._id: 64bf5732796f20daadaa7b7e
paper_authors.name: Chuang Gan
paper_title: 3D-LLM: Injecting the 3D World into Large Language Models
paper_summary: Large language models (LLMs) and Vision-Language Models (VLMs) have been
  proven to excel at multiple tasks, such as commonsense reasoning. Powerful as
  these models can be, they are not grounded in the 3D physical world, which
  involves richer concepts such as spatial relationships, affordances, physics,
  layout, and so on. In this work, we propose to inject the 3D world into large
  language models and introduce a whole new family of 3D-LLMs. Specifically,
  3D-LLMs can take 3D point clouds and their features as input and perform a
  diverse set of 3D-related tasks, including captioning, dense captioning, 3D
  question answering, task decomposition, 3D grounding, 3D-assisted dialog,
  navigation, and so on. Using three types of prompting mechanisms that we
  design, we are able to collect over 300k 3D-language data covering these tasks.
  To efficiently train 3D-LLMs, we first utilize a 3D feature extractor that
  obtains 3D features from rendered multi- view images. Then, we use 2D VLMs as
  our backbones to train our 3D-LLMs. By introducing a 3D localization mechanism,
  3D-LLMs can better capture 3D spatial information. Experiments on ScanQA show
  that our model outperforms state-of-the-art baselines by a large margin (e.g.,
  the BLEU-1 score surpasses state-of-the-art score by 9%). Furthermore,
  experiments on our held-in datasets for 3D captioning, task composition, and
  3D-assisted dialogue show that our model outperforms 2D VLMs. Qualitative
  examples also show that our model could perform more tasks beyond the scope of
  existing LLMs and VLMs. Project Page: : https://vis-www.cs.umass.edu/3dllm/.
paper_ai_summary: A new family of 3D-LLMs is introduced to perform 3D-related tasks by leveraging 3D point clouds and features, outperforming state-of-the-art baselines in tasks such as 3D question answering and captioning.
paper_ai_keywords: LLMs Vision-Language Models VLMs 3D world 3D point clouds 3D-related tasks captioning dense captioning 3D question answering task decomposition 3D grounding 3D-assisted dialog navigation 3D feature extractor 3D localization mechanism ScanQA BLEU-1 score 3D captioning task composition 3D-assisted dialogue