title: 3DSRBench: A Comprehensive 3D Spatial Reasoning Benchmark
summary: 3D spatial reasoning is the ability to analyze and interpret the positions,
  orientations, and spatial relationships of objects within the 3D space. This
  allows models to develop a comprehensive understanding of the 3D scene,
  enabling their applicability to a broader range of areas, such as autonomous
  navigation, robotics, and AR/VR. While large multi-modal models (LMMs) have
  achieved remarkable progress in a wide range of image and video understanding
  tasks, their capabilities to perform 3D spatial reasoning on diverse natural
  images are less studied. In this work we present the first comprehensive 3D
  spatial reasoning benchmark, 3DSRBench, with 2,772 manually annotated visual
  question-answer pairs across 12 question types. We conduct robust and thorough
  evaluation of 3D spatial reasoning capabilities by balancing the data
  distribution and adopting a novel FlipEval strategy. To further study the
  robustness of 3D spatial reasoning w.r.t. camera 3D viewpoints, our 3DSRBench
  includes two subsets with 3D spatial reasoning questions on paired images with
  common and uncommon viewpoints. We benchmark a wide range of open-sourced and
  proprietary LMMs, uncovering their limitations in various aspects of 3D
  awareness, such as height, orientation, location, and multi-object reasoning,
  as well as their degraded performance on images with uncommon camera
  viewpoints. Our 3DSRBench provide valuable findings and insights about the
  future development of LMMs with strong 3D reasoning capabilities. Our project
  page and dataset is available https://3dsrbench.github.io.
paper_authors._id: 675a65ea8e18d279de5b6a35
paper_authors.name: Wufei Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-12T09:03:03.299Z
paper_authors._id: 625f81afe1994410eef1c36a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1650426282769-noauth.jpeg
paper_authors.fullname: Wufei Ma
paper_authors.type: user
paper_authors.user: wufeim
paper_authors._id: 675a65ea8e18d279de5b6a36
paper_authors.name: Haoyu Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-19T18:04:07.186Z
paper_authors._id: 669ed7ccc13da2bbe2579038
paper_authors.avatarUrl: /avatars/f57341fe986591382fd9fcb0224c38e3.svg
paper_authors.fullname: Haoyu Chen
paper_authors.type: user
paper_authors.user: tonyc54
paper_authors._id: 675a65ea8e18d279de5b6a37
paper_authors.name: Guofeng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-12T09:02:35.026Z
paper_authors._id: 661434dd0dcdede49138cd7c
paper_authors.avatarUrl: /avatars/35f08bf08c5e9edfb3c78e280af718cb.svg
paper_authors.fullname: Guofeng Zhang
paper_authors.type: user
paper_authors.user: guofeng1123
paper_authors._id: 675a65ea8e18d279de5b6a38
paper_authors.name: Celso M de Melo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-12T09:02:28.868Z
paper_authors._id: 6504a2629310ce8c40f083c7
paper_authors.avatarUrl: /avatars/e6c6bdd868729d9f66240b6957f6215c.svg
paper_authors.fullname: Celso M de Melo
paper_authors.type: user
paper_authors.user: cdemelo
paper_authors._id: 675a65ea8e18d279de5b6a39
paper_authors.name: Alan Yuille
paper_authors._id: 675a65ea8e18d279de5b6a3a
paper_authors.name: Jieneng Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-12T09:02:17.141Z
paper_authors._id: 660c9ac4b202fcf3892f62fa
paper_authors.avatarUrl: /avatars/7314fd5f3f642096d0e37d3194f1aa7e.svg
paper_authors.fullname: Jieneng Chen
paper_authors.type: user
paper_authors.user: jienengchen
paper_title: 3DSRBench: A Comprehensive 3D Spatial Reasoning Benchmark
paper_summary: 3D spatial reasoning is the ability to analyze and interpret the positions,
  orientations, and spatial relationships of objects within the 3D space. This
  allows models to develop a comprehensive understanding of the 3D scene,
  enabling their applicability to a broader range of areas, such as autonomous
  navigation, robotics, and AR/VR. While large multi-modal models (LMMs) have
  achieved remarkable progress in a wide range of image and video understanding
  tasks, their capabilities to perform 3D spatial reasoning on diverse natural
  images are less studied. In this work we present the first comprehensive 3D
  spatial reasoning benchmark, 3DSRBench, with 2,772 manually annotated visual
  question-answer pairs across 12 question types. We conduct robust and thorough
  evaluation of 3D spatial reasoning capabilities by balancing the data
  distribution and adopting a novel FlipEval strategy. To further study the
  robustness of 3D spatial reasoning w.r.t. camera 3D viewpoints, our 3DSRBench
  includes two subsets with 3D spatial reasoning questions on paired images with
  common and uncommon viewpoints. We benchmark a wide range of open-sourced and
  proprietary LMMs, uncovering their limitations in various aspects of 3D
  awareness, such as height, orientation, location, and multi-object reasoning,
  as well as their degraded performance on images with uncommon camera
  viewpoints. Our 3DSRBench provide valuable findings and insights about the
  future development of LMMs with strong 3D reasoning capabilities. Our project
  page and dataset is available https://3dsrbench.github.io.
paper_ai_keywords: 3D spatial reasoning 3DSRBench multi-modal models (LMMs) visual question-answer pairs FlipEval strategy 3D viewpoints height orientation location multi-object reasoning