3DSRBench: A Comprehensive 3D Spatial Reasoning Benchmark
3D spatial reasoning is the ability to analyze and interpret the positions,
orientations, and spatial relationships of objects within the 3D space. This
allows models to develop a comprehensive understanding of the 3D scene,
enabling their applicability to a broader range of areas, such as autonomous
navigation, robotics, and AR/VR. While large multi-modal models (LMMs) have
achieved remarkable progress in a wide range of image and video understanding
tasks, their capabilities to perform 3D spatial reasoning on diverse natural
images are less studied. In this work we present the first comprehensive 3D
spatial reasoning benchmark, 3DSRBench, with 2,772 manually annotated visual
question-answer pairs across 12 question types. We conduct robust and thorough
evaluation of 3D spatial reasoning capabilities by balancing the data
distribution and adopting a novel FlipEval strategy. To further study the
robustness of 3D spatial reasoning w.r.t. camera 3D viewpoints, our 3DSRBench
includes two subsets with 3D spatial reasoning questions on paired images with
common and uncommon viewpoints. We benchmark a wide range of open-sourced and
proprietary LMMs, uncovering their limitations in various aspects of 3D
awareness, such as height, orientation, location, and multi-object reasoning,
as well as their degraded performance on images with uncommon camera
viewpoints. Our 3DSRBench provide valuable findings and insights about the
future development of LMMs with strong 3D reasoning capabilities. Our project
page and dataset is available https://3dsrbench.github.io.
675a65ea8e18d279de5b6a35
Wufei Ma
admin_assigned
2024-12-12T09:03:03.299Z
625f81afe1994410eef1c36a
https://cdn-avatars.huggingface.co/v1/production/uploads/1650426282769-noauth.jpeg
Wufei Ma
user
wufeim
675a65ea8e18d279de5b6a36
Haoyu Chen
claimed_verified
2024-12-19T18:04:07.186Z
669ed7ccc13da2bbe2579038
/avatars/f57341fe986591382fd9fcb0224c38e3.svg
Haoyu Chen
user
tonyc54
675a65ea8e18d279de5b6a37
Guofeng Zhang
admin_assigned
2024-12-12T09:02:35.026Z
661434dd0dcdede49138cd7c
/avatars/35f08bf08c5e9edfb3c78e280af718cb.svg
Guofeng Zhang
user
guofeng1123
675a65ea8e18d279de5b6a38
Celso M de Melo
admin_assigned
2024-12-12T09:02:28.868Z
6504a2629310ce8c40f083c7
/avatars/e6c6bdd868729d9f66240b6957f6215c.svg
Celso M de Melo
user
cdemelo
675a65ea8e18d279de5b6a39
Alan Yuille
675a65ea8e18d279de5b6a3a
Jieneng Chen
admin_assigned
2024-12-12T09:02:17.141Z
660c9ac4b202fcf3892f62fa
/avatars/7314fd5f3f642096d0e37d3194f1aa7e.svg
Jieneng Chen
user
jienengchen
3DSRBench: A Comprehensive 3D Spatial Reasoning Benchmark
3D spatial reasoning is the ability to analyze and interpret the positions,
orientations, and spatial relationships of objects within the 3D space. This
allows models to develop a comprehensive understanding of the 3D scene,
enabling their applicability to a broader range of areas, such as autonomous
navigation, robotics, and AR/VR. While large multi-modal models (LMMs) have
achieved remarkable progress in a wide range of image and video understanding
tasks, their capabilities to perform 3D spatial reasoning on diverse natural
images are less studied. In this work we present the first comprehensive 3D
spatial reasoning benchmark, 3DSRBench, with 2,772 manually annotated visual
question-answer pairs across 12 question types. We conduct robust and thorough
evaluation of 3D spatial reasoning capabilities by balancing the data
distribution and adopting a novel FlipEval strategy. To further study the
robustness of 3D spatial reasoning w.r.t. camera 3D viewpoints, our 3DSRBench
includes two subsets with 3D spatial reasoning questions on paired images with
common and uncommon viewpoints. We benchmark a wide range of open-sourced and
proprietary LMMs, uncovering their limitations in various aspects of 3D
awareness, such as height, orientation, location, and multi-object reasoning,
as well as their degraded performance on images with uncommon camera
viewpoints. Our 3DSRBench provide valuable findings and insights about the
future development of LMMs with strong 3D reasoning capabilities. Our project
page and dataset is available https://3dsrbench.github.io.
A benchmark named 3DSRBench evaluates large multi-modal models' 3D spatial reasoning abilities across diverse images and viewpoints, identifying their limitations and areas for improvement.
3D spatial reasoning 3DSRBench visual question-answer pairs FlipEval strategy LMMs height orientation location multi-object reasoning camera viewpoints