title: 4D-Bench: Benchmarking Multi-modal Large Language Models for 4D Object
    Understanding
summary: Multimodal Large Language Models (MLLMs) have demonstrated impressive 2D
  image/video understanding capabilities. However, there are no publicly
  standardized benchmarks to assess the abilities of MLLMs in understanding the
  4D objects (3D objects with temporal evolution over time). In this paper, we
  introduce 4D-Bench, the first benchmark to evaluate the capabilities of MLLMs
  in 4D object understanding, featuring tasks in 4D object Question Answering (4D
  object QA) and 4D object captioning. 4D-Bench provides 4D objects with diverse
  categories, high-quality annotations, and tasks necessitating multi-view
  spatial-temporal understanding, different from existing 2D image/video-based
  benchmarks. With 4D-Bench, we evaluate a wide range of open-source and
  closed-source MLLMs. The results from the 4D object captioning experiment
  indicate that MLLMs generally exhibit weaker temporal understanding compared to
  their appearance understanding, notably, while open-source models approach
  closed-source performance in appearance understanding, they show larger
  performance gaps in temporal understanding. 4D object QA yields surprising
  findings: even with simple single-object videos, MLLMs perform poorly, with
  state-of-the-art GPT-4o achieving only 63\% accuracy compared to the human
  baseline of 91\%. These findings highlight a substantial gap in 4D object
  understanding and the need for further advancements in MLLMs.
paper_authors._id: 67e2b7beb408962c5815c52d
paper_authors.name: Wenxuan Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-28T14:22:33.321Z
paper_authors._id: 658167e4499fbe1b9541adb9
paper_authors.avatarUrl: /avatars/0cec32b67c31b2d17b86f5a498400a17.svg
paper_authors.fullname: Wenxuan Zhu
paper_authors.type: user
paper_authors.user: vxuanz
paper_authors._id: 67e2b7beb408962c5815c52e
paper_authors.name: Bing Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-31T08:15:06.874Z
paper_authors._id: 666ddb45c0f3d5afc27e85ba
paper_authors.avatarUrl: /avatars/dce98fc77bd8cf9e348f2d91bc3c0225.svg
paper_authors.fullname: Bing Li
paper_authors.type: user
paper_authors.user: bing-li-ai
paper_authors._id: 67e2b7beb408962c5815c52f
paper_authors.name: Cheng Zheng
paper_authors._id: 67e2b7beb408962c5815c530
paper_authors.name: Jinjie Mai
paper_authors._id: 67e2b7beb408962c5815c531
paper_authors.name: Jun Chen
paper_authors._id: 67e2b7beb408962c5815c532
paper_authors.name: Letian Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-31T09:57:38.643Z
paper_authors._id: 67ea63f26da1353351989746
paper_authors.avatarUrl: /avatars/c7766b43f082bc71623a8fc1a23768ff.svg
paper_authors.fullname: Letian Jiang
paper_authors.type: user
paper_authors.user: TonNew
paper_authors._id: 67e2b7beb408962c5815c533
paper_authors.name: Abdullah Hamdi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-31T09:57:44.416Z
paper_authors._id: 62fe3442e9061c0170d06e0b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1660827186084-62fe3442e9061c0170d06e0b.png
paper_authors.fullname: Abdullah Hamdi
paper_authors.type: user
paper_authors.user: ajhamdi
paper_authors._id: 67e2b7beb408962c5815c534
paper_authors.name: Sara Rojas Martinez
paper_authors._id: 67e2b7beb408962c5815c535
paper_authors.name: Chia-Wen Lin
paper_authors._id: 67e2b7beb408962c5815c536
paper_authors.name: Mohamed Elhoseiny
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-31T09:58:09.462Z
paper_authors._id: 64a27d39649b0d08ca0a4ca6
paper_authors.avatarUrl: /avatars/e4446a875506c10de9ae28411dc6416d.svg
paper_authors.fullname: Mohamed Elhoseiny
paper_authors.type: user
paper_authors.user: mhelhoseiny
paper_authors._id: 67e2b7beb408962c5815c537
paper_authors.name: Bernard Ghanem
paper_title: 4D-Bench: Benchmarking Multi-modal Large Language Models for 4D Object
    Understanding
paper_summary: Multimodal Large Language Models (MLLMs) have demonstrated impressive 2D
  image/video understanding capabilities. However, there are no publicly
  standardized benchmarks to assess the abilities of MLLMs in understanding the
  4D objects (3D objects with temporal evolution over time). In this paper, we
  introduce 4D-Bench, the first benchmark to evaluate the capabilities of MLLMs
  in 4D object understanding, featuring tasks in 4D object Question Answering (4D
  object QA) and 4D object captioning. 4D-Bench provides 4D objects with diverse
  categories, high-quality annotations, and tasks necessitating multi-view
  spatial-temporal understanding, different from existing 2D image/video-based
  benchmarks. With 4D-Bench, we evaluate a wide range of open-source and
  closed-source MLLMs. The results from the 4D object captioning experiment
  indicate that MLLMs generally exhibit weaker temporal understanding compared to
  their appearance understanding, notably, while open-source models approach
  closed-source performance in appearance understanding, they show larger
  performance gaps in temporal understanding. 4D object QA yields surprising
  findings: even with simple single-object videos, MLLMs perform poorly, with
  state-of-the-art GPT-4o achieving only 63\% accuracy compared to the human
  baseline of 91\%. These findings highlight a substantial gap in 4D object
  understanding and the need for further advancements in MLLMs.
paper_ai_summary: 4D-Bench is introduced as the first benchmark to evaluate MLLMs on 4D object understanding, revealing weaknesses in temporal understanding compared to appearance.
paper_ai_keywords: Multimodal Large Language Models MLLMs 4D object understanding 4D-Bench 4D object Question Answering 4D object captioning multi-view spatial-temporal understanding GPT-4o