title: Can MLLMs Guide Me Home? A Benchmark Study on Fine-Grained Visual
    Reasoning from Transit Maps
summary: Multimodal large language models (MLLMs) have recently achieved significant
  progress in visual tasks, including semantic scene understanding and text-image
  alignment, with reasoning variants enhancing performance on complex tasks
  involving mathematics and logic. However, their capacity for reasoning tasks
  involving fine-grained visual understanding remains insufficiently evaluated.
  To address this gap, we introduce ReasonMap, a benchmark designed to assess the
  fine-grained visual understanding and spatial reasoning abilities of MLLMs.
  ReasonMap encompasses high-resolution transit maps from 30 cities across 13
  countries and includes 1,008 question-answer pairs spanning two question types
  and three templates. Furthermore, we design a two-level evaluation pipeline
  that properly assesses answer correctness and quality. Comprehensive
  evaluations of 15 popular MLLMs, including both base and reasoning variants,
  reveal a counterintuitive pattern: among open-source models, base models
  outperform reasoning ones, while the opposite trend is observed in
  closed-source models. Additionally, performance generally degrades when visual
  inputs are masked, indicating that while MLLMs can leverage prior knowledge to
  answer some questions, fine-grained visual reasoning tasks still require
  genuine visual perception for strong performance. Our benchmark study offers
  new insights into visual reasoning and contributes to investigating the gap
  between open-source and closed-source models.
paper_authors._id: 68351dde0c0aff775f3933ee
paper_authors.name: Sicheng Feng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T07:52:13.406Z
paper_authors._id: 67a4a26d5e65aa63c6d30e68
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/67a4a26d5e65aa63c6d30e68/GtodlJGw-_IL2DTXQTucz.jpeg
paper_authors.fullname: Sicheng Feng
paper_authors.type: user
paper_authors.user: FSCCS
paper_authors._id: 68351dde0c0aff775f3933ef
paper_authors.name: Song Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T20:18:42.487Z
paper_authors._id: 66863d26e2b71e3d09189ae9
paper_authors.avatarUrl: /avatars/3c0e6f30e053f2e622ae75e1dc43edba.svg
paper_authors.fullname: Song Wang
paper_authors.type: user
paper_authors.user: songw-zju
paper_authors._id: 68351dde0c0aff775f3933f0
paper_authors.name: Shuyi Ouyang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T12:35:44.568Z
paper_authors._id: 677d580857defdfa3cf5a719
paper_authors.avatarUrl: /avatars/4cdf68e97eab8b7da951b911e466ee23.svg
paper_authors.fullname: Shuyi Ouyang
paper_authors.type: user
paper_authors.user: NaturalKnight
paper_authors._id: 68351dde0c0aff775f3933f1
paper_authors.name: Lingdong Kong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T12:35:38.601Z
paper_authors._id: 62df78222d89ce551ce0f71d
paper_authors.avatarUrl: /avatars/89fba294cff2d2f941d121c1923e4c76.svg
paper_authors.fullname: Lingdong Kong
paper_authors.type: user
paper_authors.user: ldkong
paper_authors._id: 68351dde0c0aff775f3933f2
paper_authors.name: Zikai Song
paper_authors._id: 68351dde0c0aff775f3933f3
paper_authors.name: Jianke Zhu
paper_authors._id: 68351dde0c0aff775f3933f4
paper_authors.name: Huan Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T07:52:11.085Z
paper_authors._id: 62b624f3b52bef716e248fd7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b624f3b52bef716e248fd7/AllcccKH-eBWduA8KVnOQ.png
paper_authors.fullname: Huan Wang
paper_authors.type: user
paper_authors.user: Huan-WhoRegisteredMyName
paper_authors._id: 68351dde0c0aff775f3933f5
paper_authors.name: Xinchao Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T12:35:25.108Z
paper_authors._id: 63fc03a50aab060792ffef39
paper_authors.avatarUrl: /avatars/9d5b1bb2a41928e08176b703935133ab.svg
paper_authors.fullname: Wangxinchao
paper_authors.type: user
paper_authors.user: wxcTest
paper_title: Can MLLMs Guide Me Home? A Benchmark Study on Fine-Grained Visual
    Reasoning from Transit Maps
paper_summary: Multimodal large language models (MLLMs) have recently achieved significant
  progress in visual tasks, including semantic scene understanding and text-image
  alignment, with reasoning variants enhancing performance on complex tasks
  involving mathematics and logic. However, their capacity for reasoning tasks
  involving fine-grained visual understanding remains insufficiently evaluated.
  To address this gap, we introduce ReasonMap, a benchmark designed to assess the
  fine-grained visual understanding and spatial reasoning abilities of MLLMs.
  ReasonMap encompasses high-resolution transit maps from 30 cities across 13
  countries and includes 1,008 question-answer pairs spanning two question types
  and three templates. Furthermore, we design a two-level evaluation pipeline
  that properly assesses answer correctness and quality. Comprehensive
  evaluations of 15 popular MLLMs, including both base and reasoning variants,
  reveal a counterintuitive pattern: among open-source models, base models
  outperform reasoning ones, while the opposite trend is observed in
  closed-source models. Additionally, performance generally degrades when visual
  inputs are masked, indicating that while MLLMs can leverage prior knowledge to
  answer some questions, fine-grained visual reasoning tasks still require
  genuine visual perception for strong performance. Our benchmark study offers
  new insights into visual reasoning and contributes to investigating the gap
  between open-source and closed-source models.
paper_ai_summary: ReasonMap evaluates the fine-grained visual understanding and spatial reasoning abilities of multimodal large language models, revealing that base models often outperform reasoning variants and highlighting the importance of genuine visual perception for complex tasks.
paper_ai_keywords: multimodal large language models MLLMs semantic scene understanding text-image alignment reasoning variants ReasonMap high-resolution transit maps question-answer pairs two-level evaluation pipeline open-source models closed-source models visual reasoning