title: M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts
summary: Recently, 3D understanding has become popular to facilitate autonomous agents
  to perform further decisionmaking. However, existing 3D datasets and methods
  are often limited to specific tasks. On the other hand, recent progress in
  Large Language Models (LLMs) and Multimodal Language Models (MLMs) have
  demonstrated exceptional general language and imagery tasking performance.
  Therefore, it is interesting to unlock MLM's potential to be 3D generalist for
  wider tasks. However, current MLMs' research has been less focused on 3D tasks
  due to a lack of large-scale 3D instruction-following datasets. In this work,
  we introduce a comprehensive 3D instructionfollowing dataset called M3DBench,
  which possesses the following characteristics: 1) It supports general
  multimodal instructions interleaved with text, images, 3D objects, and other
  visual prompts. 2) It unifies diverse 3D tasks at both region and scene levels,
  covering a variety of fundamental abilities in real-world 3D environments. 3)
  It is a large-scale 3D instruction-following dataset with over 320k
  instruction-response pairs. Furthermore, we establish a new benchmark for
  assessing the performance of large models in understanding multi-modal 3D
  prompts. Extensive experiments demonstrate the effectiveness of our dataset and
  baseline, supporting general 3D-centric tasks, which can inspire future
  research.
paper_authors._id: 65812e46035c028f333d78f6
paper_authors.name: Mingsheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-19T08:40:13.216Z
paper_authors._id: 64d7b8c346bda7d8f50cb52f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64d7b8c346bda7d8f50cb52f/XjCHm-Jmy4T5cdDUVgGS1.jpeg
paper_authors.fullname: limingsheng
paper_authors.type: user
paper_authors.user: MSheng-Lee
paper_authors._id: 65812e46035c028f333d78f7
paper_authors.name: Xin Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-20T08:19:49.887Z
paper_authors._id: 63b543b2bda8d44adf2d7d0a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63b543b2bda8d44adf2d7d0a/G_AzO8iA-WwEcxvJEKVQ0.jpeg
paper_authors.fullname: Chen
paper_authors.type: user
paper_authors.user: ChrisChenXin
paper_authors._id: 65812e46035c028f333d78f8
paper_authors.name: Chi Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-19T08:55:37.617Z
paper_authors._id: 64196320ed725fef64419c2a
paper_authors.avatarUrl: /avatars/96feb22fb5e8931d6c9e0ea06148266f.svg
paper_authors.fullname: Chi Zhang
paper_authors.type: user
paper_authors.user: DrChiZhang
paper_authors._id: 65812e46035c028f333d78f9
paper_authors.name: Sijin Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-19T08:43:59.603Z
paper_authors._id: 64d6ee19c3b41747a271da59
paper_authors.avatarUrl: /avatars/c67da5d25fdf1dcb552772500955edb2.svg
paper_authors.fullname: Sijin Chen
paper_authors.type: user
paper_authors.user: ch3cook-fdu
paper_authors._id: 65812e46035c028f333d78fa
paper_authors.name: Hongyuan Zhu
paper_authors._id: 65812e46035c028f333d78fb
paper_authors.name: Fukun Yin
paper_authors._id: 65812e46035c028f333d78fc
paper_authors.name: Gang Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-19T08:44:54.293Z
paper_authors._id: 63417332c5565a4b8d43a0d8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63417332c5565a4b8d43a0d8/MmnYG7Wu2Z_lqCBvTfJmy.png
paper_authors.fullname: Gang Yu
paper_authors.type: user
paper_authors.user: skicy
paper_authors._id: 65812e46035c028f333d78fd
paper_authors.name: Tao Chen
paper_title: M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts
paper_summary: Recently, 3D understanding has become popular to facilitate autonomous agents
  to perform further decisionmaking. However, existing 3D datasets and methods
  are often limited to specific tasks. On the other hand, recent progress in
  Large Language Models (LLMs) and Multimodal Language Models (MLMs) have
  demonstrated exceptional general language and imagery tasking performance.
  Therefore, it is interesting to unlock MLM's potential to be 3D generalist for
  wider tasks. However, current MLMs' research has been less focused on 3D tasks
  due to a lack of large-scale 3D instruction-following datasets. In this work,
  we introduce a comprehensive 3D instructionfollowing dataset called M3DBench,
  which possesses the following characteristics: 1) It supports general
  multimodal instructions interleaved with text, images, 3D objects, and other
  visual prompts. 2) It unifies diverse 3D tasks at both region and scene levels,
  covering a variety of fundamental abilities in real-world 3D environments. 3)
  It is a large-scale 3D instruction-following dataset with over 320k
  instruction-response pairs. Furthermore, we establish a new benchmark for
  assessing the performance of large models in understanding multi-modal 3D
  prompts. Extensive experiments demonstrate the effectiveness of our dataset and
  baseline, supporting general 3D-centric tasks, which can inspire future
  research.
paper_ai_summary: M3DBench is a comprehensive 3D instruction-following dataset enabling a broad range of multimodal 3D tasks and serves as a new benchmark for assessing model performance in understanding 3D prompts.
paper_ai_keywords: Large Language Models Multimodal Language Models M3DBench instruction-following dataset multimodal instructions 3D tasks region level scene level instruction-response pairs benchmark