M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts
Recently, 3D understanding has become popular to facilitate autonomous agents
to perform further decisionmaking. However, existing 3D datasets and methods
are often limited to specific tasks. On the other hand, recent progress in
Large Language Models (LLMs) and Multimodal Language Models (MLMs) have
demonstrated exceptional general language and imagery tasking performance.
Therefore, it is interesting to unlock MLM's potential to be 3D generalist for
wider tasks. However, current MLMs' research has been less focused on 3D tasks
due to a lack of large-scale 3D instruction-following datasets. In this work,
we introduce a comprehensive 3D instructionfollowing dataset called M3DBench,
which possesses the following characteristics: 1) It supports general
multimodal instructions interleaved with text, images, 3D objects, and other
visual prompts. 2) It unifies diverse 3D tasks at both region and scene levels,
covering a variety of fundamental abilities in real-world 3D environments. 3)
It is a large-scale 3D instruction-following dataset with over 320k
instruction-response pairs. Furthermore, we establish a new benchmark for
assessing the performance of large models in understanding multi-modal 3D
prompts. Extensive experiments demonstrate the effectiveness of our dataset and
baseline, supporting general 3D-centric tasks, which can inspire future
research.
65812e46035c028f333d78f6
Mingsheng Li
admin_assigned
2023-12-19T08:40:13.216Z
64d7b8c346bda7d8f50cb52f
https://cdn-avatars.huggingface.co/v1/production/uploads/64d7b8c346bda7d8f50cb52f/XjCHm-Jmy4T5cdDUVgGS1.jpeg
limingsheng
user
MSheng-Lee
65812e46035c028f333d78f7
Xin Chen
claimed_verified
2023-12-20T08:19:49.887Z
63b543b2bda8d44adf2d7d0a
https://cdn-avatars.huggingface.co/v1/production/uploads/63b543b2bda8d44adf2d7d0a/G_AzO8iA-WwEcxvJEKVQ0.jpeg
Chen
user
ChrisChenXin
65812e46035c028f333d78f8
Chi Zhang
claimed_verified
2023-12-19T08:55:37.617Z
64196320ed725fef64419c2a
/avatars/96feb22fb5e8931d6c9e0ea06148266f.svg
Chi Zhang
user
DrChiZhang
65812e46035c028f333d78f9
Sijin Chen
admin_assigned
2023-12-19T08:43:59.603Z
64d6ee19c3b41747a271da59
/avatars/c67da5d25fdf1dcb552772500955edb2.svg
Sijin Chen
user
ch3cook-fdu
65812e46035c028f333d78fa
Hongyuan Zhu
65812e46035c028f333d78fb
Fukun Yin
65812e46035c028f333d78fc
Gang Yu
admin_assigned
2023-12-19T08:44:54.293Z
63417332c5565a4b8d43a0d8
https://cdn-avatars.huggingface.co/v1/production/uploads/63417332c5565a4b8d43a0d8/MmnYG7Wu2Z_lqCBvTfJmy.png
Gang Yu
user
skicy
65812e46035c028f333d78fd
Tao Chen
M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts
Recently, 3D understanding has become popular to facilitate autonomous agents
to perform further decisionmaking. However, existing 3D datasets and methods
are often limited to specific tasks. On the other hand, recent progress in
Large Language Models (LLMs) and Multimodal Language Models (MLMs) have
demonstrated exceptional general language and imagery tasking performance.
Therefore, it is interesting to unlock MLM's potential to be 3D generalist for
wider tasks. However, current MLMs' research has been less focused on 3D tasks
due to a lack of large-scale 3D instruction-following datasets. In this work,
we introduce a comprehensive 3D instructionfollowing dataset called M3DBench,
which possesses the following characteristics: 1) It supports general
multimodal instructions interleaved with text, images, 3D objects, and other
visual prompts. 2) It unifies diverse 3D tasks at both region and scene levels,
covering a variety of fundamental abilities in real-world 3D environments. 3)
It is a large-scale 3D instruction-following dataset with over 320k
instruction-response pairs. Furthermore, we establish a new benchmark for
assessing the performance of large models in understanding multi-modal 3D
prompts. Extensive experiments demonstrate the effectiveness of our dataset and
baseline, supporting general 3D-centric tasks, which can inspire future
research.
M3DBench is a comprehensive 3D instruction-following dataset enabling a broad range of multimodal 3D tasks and serves as a new benchmark for assessing model performance in understanding 3D prompts.
Large Language Models Multimodal Language Models M3DBench instruction-following dataset multimodal instructions 3D tasks region level scene level instruction-response pairs benchmark