title: Right Side Up? Disentangling Orientation Understanding in MLLMs with
    Fine-grained Multi-axis Perception Tasks
summary: Object orientation understanding represents a fundamental challenge in visual
  perception critical for applications like robotic manipulation and augmented
  reality. Current vision-language benchmarks fail to isolate this capability,
  often conflating it with positional relationships and general scene
  understanding. We introduce DORI (Discriminative Orientation Reasoning
  Intelligence), a comprehensive benchmark establishing object orientation
  perception as a primary evaluation target. DORI assesses four dimensions of
  orientation comprehension: frontal alignment, rotational transformations,
  relative directional relationships, and canonical orientation understanding.
  Through carefully curated tasks from 11 datasets spanning 67 object categories
  across synthetic and real-world scenarios, DORI provides insights on how
  multi-modal systems understand object orientations. Our evaluation of 15
  state-of-the-art vision-language models reveals critical limitations: even the
  best models achieve only 54.2% accuracy on coarse tasks and 33.0% on granular
  orientation judgments, with performance deteriorating for tasks requiring
  reference frame shifts or compound rotations. These findings demonstrate the
  need for dedicated orientation representation mechanisms, as models show
  systematic inability to perform precise angular estimations, track orientation
  changes across viewpoints, and understand compound rotations - suggesting
  limitations in their internal 3D spatial representations. As the first
  diagnostic framework specifically designed for orientation awareness in
  multimodal systems, DORI offers implications for improving robotic control, 3D
  scene reconstruction, and human-AI interaction in physical environments. DORI
  data: https://huggingface.co/datasets/appledora/DORI-Benchmark
paper_authors._id: 68386daaa2d6f83cf1fb97a2
paper_authors.name: Keanu Nichols
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T20:19:27.650Z
paper_authors._id: 64505b4a5fdcff143ad8593c
paper_authors.avatarUrl: /avatars/1278a3f4c8c7be342bb6ab7ce4952ab7.svg
paper_authors.fullname: Keanu Nichols
paper_authors.type: user
paper_authors.user: kmn5409
paper_authors._id: 68386daaa2d6f83cf1fb97a3
paper_authors.name: Nazia Tasnim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-30T06:54:49.440Z
paper_authors._id: 6199e56f90692f0d92388482
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1637475683741-noauth.jpeg
paper_authors.fullname: Nazia Tasnim
paper_authors.type: user
paper_authors.user: appledora
paper_authors._id: 68386daaa2d6f83cf1fb97a4
paper_authors.name: Yan Yuting
paper_authors._id: 68386daaa2d6f83cf1fb97a5
paper_authors.name: Nicholas Ikechukwu
paper_authors._id: 68386daaa2d6f83cf1fb97a6
paper_authors.name: Elva Zou
paper_authors._id: 68386daaa2d6f83cf1fb97a7
paper_authors.name: Deepti Ghadiyaram
paper_authors._id: 68386daaa2d6f83cf1fb97a8
paper_authors.name: Bryan Plummer
paper_title: Right Side Up? Disentangling Orientation Understanding in MLLMs with
    Fine-grained Multi-axis Perception Tasks
paper_summary: Object orientation understanding represents a fundamental challenge in visual
  perception critical for applications like robotic manipulation and augmented
  reality. Current vision-language benchmarks fail to isolate this capability,
  often conflating it with positional relationships and general scene
  understanding. We introduce DORI (Discriminative Orientation Reasoning
  Intelligence), a comprehensive benchmark establishing object orientation
  perception as a primary evaluation target. DORI assesses four dimensions of
  orientation comprehension: frontal alignment, rotational transformations,
  relative directional relationships, and canonical orientation understanding.
  Through carefully curated tasks from 11 datasets spanning 67 object categories
  across synthetic and real-world scenarios, DORI provides insights on how
  multi-modal systems understand object orientations. Our evaluation of 15
  state-of-the-art vision-language models reveals critical limitations: even the
  best models achieve only 54.2% accuracy on coarse tasks and 33.0% on granular
  orientation judgments, with performance deteriorating for tasks requiring
  reference frame shifts or compound rotations. These findings demonstrate the
  need for dedicated orientation representation mechanisms, as models show
  systematic inability to perform precise angular estimations, track orientation
  changes across viewpoints, and understand compound rotations - suggesting
  limitations in their internal 3D spatial representations. As the first
  diagnostic framework specifically designed for orientation awareness in
  multimodal systems, DORI offers implications for improving robotic control, 3D
  scene reconstruction, and human-AI interaction in physical environments. DORI
  data: https://huggingface.co/datasets/appledora/DORI-Benchmark
paper_ai_summary: DORI evaluates object orientation perception in vision-language models, revealing critical limitations in their ability to accurately estimate and track orientations, particularly with reference frame shifts and compound rotations.
paper_ai_keywords: discriminative orientation reasoning intelligence orientation comprehension frontal alignment rotational transformations relative directional relationships canonical orientation understanding vision-language models angular estimations 3D spatial representations