Right Side Up? Disentangling Orientation Understanding in MLLMs with
  Fine-grained Multi-axis Perception Tasks
Object orientation understanding represents a fundamental challenge in visual
perception critical for applications like robotic manipulation and augmented
reality. Current vision-language benchmarks fail to isolate this capability,
often conflating it with positional relationships and general scene
understanding. We introduce DORI (Discriminative Orientation Reasoning
Intelligence), a comprehensive benchmark establishing object orientation
perception as a primary evaluation target. DORI assesses four dimensions of
orientation comprehension: frontal alignment, rotational transformations,
relative directional relationships, and canonical orientation understanding.
Through carefully curated tasks from 11 datasets spanning 67 object categories
across synthetic and real-world scenarios, DORI provides insights on how
multi-modal systems understand object orientations. Our evaluation of 15
state-of-the-art vision-language models reveals critical limitations: even the
best models achieve only 54.2% accuracy on coarse tasks and 33.0% on granular
orientation judgments, with performance deteriorating for tasks requiring
reference frame shifts or compound rotations. These findings demonstrate the
need for dedicated orientation representation mechanisms, as models show
systematic inability to perform precise angular estimations, track orientation
changes across viewpoints, and understand compound rotations - suggesting
limitations in their internal 3D spatial representations. As the first
diagnostic framework specifically designed for orientation awareness in
multimodal systems, DORI offers implications for improving robotic control, 3D
scene reconstruction, and human-AI interaction in physical environments. DORI
data: https://huggingface.co/datasets/appledora/DORI-Benchmark
68386daaa2d6f83cf1fb97a2
Keanu Nichols
claimed_verified
2025-05-29T20:19:27.650Z
64505b4a5fdcff143ad8593c
/avatars/1278a3f4c8c7be342bb6ab7ce4952ab7.svg
Keanu Nichols
user
kmn5409
68386daaa2d6f83cf1fb97a3
Nazia Tasnim
claimed_verified
2025-05-30T06:54:49.440Z
6199e56f90692f0d92388482
https://cdn-avatars.huggingface.co/v1/production/uploads/1637475683741-noauth.jpeg
Nazia Tasnim
user
appledora
68386daaa2d6f83cf1fb97a4
Yan Yuting
68386daaa2d6f83cf1fb97a5
Nicholas Ikechukwu
68386daaa2d6f83cf1fb97a6
Elva Zou
68386daaa2d6f83cf1fb97a7
Deepti Ghadiyaram
68386daaa2d6f83cf1fb97a8
Bryan Plummer
Right Side Up? Disentangling Orientation Understanding in MLLMs with
  Fine-grained Multi-axis Perception Tasks
Object orientation understanding represents a fundamental challenge in visual
perception critical for applications like robotic manipulation and augmented
reality. Current vision-language benchmarks fail to isolate this capability,
often conflating it with positional relationships and general scene
understanding. We introduce DORI (Discriminative Orientation Reasoning
Intelligence), a comprehensive benchmark establishing object orientation
perception as a primary evaluation target. DORI assesses four dimensions of
orientation comprehension: frontal alignment, rotational transformations,
relative directional relationships, and canonical orientation understanding.
Through carefully curated tasks from 11 datasets spanning 67 object categories
across synthetic and real-world scenarios, DORI provides insights on how
multi-modal systems understand object orientations. Our evaluation of 15
state-of-the-art vision-language models reveals critical limitations: even the
best models achieve only 54.2% accuracy on coarse tasks and 33.0% on granular
orientation judgments, with performance deteriorating for tasks requiring
reference frame shifts or compound rotations. These findings demonstrate the
need for dedicated orientation representation mechanisms, as models show
systematic inability to perform precise angular estimations, track orientation
changes across viewpoints, and understand compound rotations - suggesting
limitations in their internal 3D spatial representations. As the first
diagnostic framework specifically designed for orientation awareness in
multimodal systems, DORI offers implications for improving robotic control, 3D
scene reconstruction, and human-AI interaction in physical environments. DORI
data: https://huggingface.co/datasets/appledora/DORI-Benchmark
DORI evaluates object orientation perception in vision-language models, revealing critical limitations in their ability to accurately estimate and track orientations, particularly with reference frame shifts and compound rotations.
discriminative orientation reasoning intelligence orientation comprehension frontal alignment rotational transformations relative directional relationships canonical orientation understanding vision-language models angular estimations 3D spatial representations