4D-Bench: Benchmarking Multi-modal Large Language Models for 4D Object
  Understanding
Multimodal Large Language Models (MLLMs) have demonstrated impressive 2D
image/video understanding capabilities. However, there are no publicly
standardized benchmarks to assess the abilities of MLLMs in understanding the
4D objects (3D objects with temporal evolution over time). In this paper, we
introduce 4D-Bench, the first benchmark to evaluate the capabilities of MLLMs
in 4D object understanding, featuring tasks in 4D object Question Answering (4D
object QA) and 4D object captioning. 4D-Bench provides 4D objects with diverse
categories, high-quality annotations, and tasks necessitating multi-view
spatial-temporal understanding, different from existing 2D image/video-based
benchmarks. With 4D-Bench, we evaluate a wide range of open-source and
closed-source MLLMs. The results from the 4D object captioning experiment
indicate that MLLMs generally exhibit weaker temporal understanding compared to
their appearance understanding, notably, while open-source models approach
closed-source performance in appearance understanding, they show larger
performance gaps in temporal understanding. 4D object QA yields surprising
findings: even with simple single-object videos, MLLMs perform poorly, with
state-of-the-art GPT-4o achieving only 63\% accuracy compared to the human
baseline of 91\%. These findings highlight a substantial gap in 4D object
understanding and the need for further advancements in MLLMs.
67e2b7beb408962c5815c52d
Wenxuan Zhu
claimed_verified
2025-03-28T14:22:33.321Z
658167e4499fbe1b9541adb9
/avatars/0cec32b67c31b2d17b86f5a498400a17.svg
Wenxuan Zhu
user
vxuanz
67e2b7beb408962c5815c52e
Bing Li
claimed_verified
2025-03-31T08:15:06.874Z
666ddb45c0f3d5afc27e85ba
/avatars/dce98fc77bd8cf9e348f2d91bc3c0225.svg
Bing Li
user
bing-li-ai
67e2b7beb408962c5815c52f
Cheng Zheng
67e2b7beb408962c5815c530
Jinjie Mai
67e2b7beb408962c5815c531
Jun Chen
67e2b7beb408962c5815c532
Letian Jiang
admin_assigned
2025-03-31T09:57:38.643Z
67ea63f26da1353351989746
/avatars/c7766b43f082bc71623a8fc1a23768ff.svg
Letian Jiang
user
TonNew
67e2b7beb408962c5815c533
Abdullah Hamdi
admin_assigned
2025-03-31T09:57:44.416Z
62fe3442e9061c0170d06e0b
https://cdn-avatars.huggingface.co/v1/production/uploads/1660827186084-62fe3442e9061c0170d06e0b.png
Abdullah Hamdi
user
ajhamdi
67e2b7beb408962c5815c534
Sara Rojas Martinez
67e2b7beb408962c5815c535
Chia-Wen Lin
67e2b7beb408962c5815c536
Mohamed Elhoseiny
admin_assigned
2025-03-31T09:58:09.462Z
64a27d39649b0d08ca0a4ca6
/avatars/e4446a875506c10de9ae28411dc6416d.svg
Mohamed Elhoseiny
user
mhelhoseiny
67e2b7beb408962c5815c537
Bernard Ghanem
4D-Bench: Benchmarking Multi-modal Large Language Models for 4D Object
  Understanding
Multimodal Large Language Models (MLLMs) have demonstrated impressive 2D
image/video understanding capabilities. However, there are no publicly
standardized benchmarks to assess the abilities of MLLMs in understanding the
4D objects (3D objects with temporal evolution over time). In this paper, we
introduce 4D-Bench, the first benchmark to evaluate the capabilities of MLLMs
in 4D object understanding, featuring tasks in 4D object Question Answering (4D
object QA) and 4D object captioning. 4D-Bench provides 4D objects with diverse
categories, high-quality annotations, and tasks necessitating multi-view
spatial-temporal understanding, different from existing 2D image/video-based
benchmarks. With 4D-Bench, we evaluate a wide range of open-source and
closed-source MLLMs. The results from the 4D object captioning experiment
indicate that MLLMs generally exhibit weaker temporal understanding compared to
their appearance understanding, notably, while open-source models approach
closed-source performance in appearance understanding, they show larger
performance gaps in temporal understanding. 4D object QA yields surprising
findings: even with simple single-object videos, MLLMs perform poorly, with
state-of-the-art GPT-4o achieving only 63\% accuracy compared to the human
baseline of 91\%. These findings highlight a substantial gap in 4D object
understanding and the need for further advancements in MLLMs.
4D-Bench is introduced as the first benchmark to evaluate MLLMs on 4D object understanding, revealing weaknesses in temporal understanding compared to appearance.
Multimodal Large Language Models MLLMs 4D object understanding 4D-Bench 4D object Question Answering 4D object captioning multi-view spatial-temporal understanding GPT-4o