Can MLLMs Guide Me Home? A Benchmark Study on Fine-Grained Visual
  Reasoning from Transit Maps
Multimodal large language models (MLLMs) have recently achieved significant
progress in visual tasks, including semantic scene understanding and text-image
alignment, with reasoning variants enhancing performance on complex tasks
involving mathematics and logic. However, their capacity for reasoning tasks
involving fine-grained visual understanding remains insufficiently evaluated.
To address this gap, we introduce ReasonMap, a benchmark designed to assess the
fine-grained visual understanding and spatial reasoning abilities of MLLMs.
ReasonMap encompasses high-resolution transit maps from 30 cities across 13
countries and includes 1,008 question-answer pairs spanning two question types
and three templates. Furthermore, we design a two-level evaluation pipeline
that properly assesses answer correctness and quality. Comprehensive
evaluations of 15 popular MLLMs, including both base and reasoning variants,
reveal a counterintuitive pattern: among open-source models, base models
outperform reasoning ones, while the opposite trend is observed in
closed-source models. Additionally, performance generally degrades when visual
inputs are masked, indicating that while MLLMs can leverage prior knowledge to
answer some questions, fine-grained visual reasoning tasks still require
genuine visual perception for strong performance. Our benchmark study offers
new insights into visual reasoning and contributes to investigating the gap
between open-source and closed-source models.
68351dde0c0aff775f3933ee
Sicheng Feng
claimed_verified
2025-05-27T07:52:13.406Z
67a4a26d5e65aa63c6d30e68
https://cdn-avatars.huggingface.co/v1/production/uploads/67a4a26d5e65aa63c6d30e68/GtodlJGw-_IL2DTXQTucz.jpeg
Sicheng Feng
user
FSCCS
68351dde0c0aff775f3933ef
Song Wang
claimed_verified
2025-05-29T20:18:42.487Z
66863d26e2b71e3d09189ae9
/avatars/3c0e6f30e053f2e622ae75e1dc43edba.svg
Song Wang
user
songw-zju
68351dde0c0aff775f3933f0
Shuyi Ouyang
admin_assigned
2025-05-27T12:35:44.568Z
677d580857defdfa3cf5a719
/avatars/4cdf68e97eab8b7da951b911e466ee23.svg
Shuyi Ouyang
user
NaturalKnight
68351dde0c0aff775f3933f1
Lingdong Kong
admin_assigned
2025-05-27T12:35:38.601Z
62df78222d89ce551ce0f71d
/avatars/89fba294cff2d2f941d121c1923e4c76.svg
Lingdong Kong
user
ldkong
68351dde0c0aff775f3933f2
Zikai Song
68351dde0c0aff775f3933f3
Jianke Zhu
68351dde0c0aff775f3933f4
Huan Wang
claimed_verified
2025-05-27T07:52:11.085Z
62b624f3b52bef716e248fd7
https://cdn-avatars.huggingface.co/v1/production/uploads/62b624f3b52bef716e248fd7/AllcccKH-eBWduA8KVnOQ.png
Huan Wang
user
Huan-WhoRegisteredMyName
68351dde0c0aff775f3933f5
Xinchao Wang
admin_assigned
2025-05-27T12:35:25.108Z
63fc03a50aab060792ffef39
/avatars/9d5b1bb2a41928e08176b703935133ab.svg
Wangxinchao
user
wxcTest
Can MLLMs Guide Me Home? A Benchmark Study on Fine-Grained Visual
  Reasoning from Transit Maps
Multimodal large language models (MLLMs) have recently achieved significant
progress in visual tasks, including semantic scene understanding and text-image
alignment, with reasoning variants enhancing performance on complex tasks
involving mathematics and logic. However, their capacity for reasoning tasks
involving fine-grained visual understanding remains insufficiently evaluated.
To address this gap, we introduce ReasonMap, a benchmark designed to assess the
fine-grained visual understanding and spatial reasoning abilities of MLLMs.
ReasonMap encompasses high-resolution transit maps from 30 cities across 13
countries and includes 1,008 question-answer pairs spanning two question types
and three templates. Furthermore, we design a two-level evaluation pipeline
that properly assesses answer correctness and quality. Comprehensive
evaluations of 15 popular MLLMs, including both base and reasoning variants,
reveal a counterintuitive pattern: among open-source models, base models
outperform reasoning ones, while the opposite trend is observed in
closed-source models. Additionally, performance generally degrades when visual
inputs are masked, indicating that while MLLMs can leverage prior knowledge to
answer some questions, fine-grained visual reasoning tasks still require
genuine visual perception for strong performance. Our benchmark study offers
new insights into visual reasoning and contributes to investigating the gap
between open-source and closed-source models.
ReasonMap evaluates the fine-grained visual understanding and spatial reasoning abilities of multimodal large language models, revealing that base models often outperform reasoning variants and highlighting the importance of genuine visual perception for complex tasks.
multimodal large language models MLLMs semantic scene understanding text-image alignment reasoning variants ReasonMap high-resolution transit maps question-answer pairs two-level evaluation pipeline open-source models closed-source models visual reasoning