title: MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation
    Models
summary: Recent advancements in foundation models have enhanced AI systems'
  capabilities in autonomous tool usage and reasoning. However, their ability in
  location or map-based reasoning - which improves daily life by optimizing
  navigation, facilitating resource discovery, and streamlining logistics - has
  not been systematically studied. To bridge this gap, we introduce MapEval, a
  benchmark designed to assess diverse and complex map-based user queries with
  geo-spatial reasoning. MapEval features three task types (textual, API-based,
  and visual) that require collecting world information via map tools, processing
  heterogeneous geo-spatial contexts (e.g., named entities, travel distances,
  user reviews or ratings, images), and compositional reasoning, which all
  state-of-the-art foundation models find challenging. Comprising 700 unique
  multiple-choice questions about locations across 180 cities and 54 countries,
  MapEval evaluates foundation models' ability to handle spatial relationships,
  map infographics, travel planning, and navigation challenges. Using MapEval, we
  conducted a comprehensive evaluation of 28 prominent foundation models. While
  no single model excelled across all tasks, Claude-3.5-Sonnet, GPT-4o, and
  Gemini-1.5-Pro achieved competitive performance overall. However, substantial
  performance gaps emerged, particularly in MapEval, where agents with
  Claude-3.5-Sonnet outperformed GPT-4o and Gemini-1.5-Pro by 16% and 21%,
  respectively, and the gaps became even more amplified when compared to
  open-source LLMs. Our detailed analyses provide insights into the strengths and
  weaknesses of current models, though all models still fall short of human
  performance by more than 20% on average, struggling with complex map images and
  rigorous geo-spatial reasoning. This gap highlights MapEval's critical role in
  advancing general-purpose foundation models with stronger geo-spatial
  understanding.
paper_authors._id: 6777829e2f92b6f8edf97422
paper_authors.name: Mahir Labib Dihan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-03T14:03:04.107Z
paper_authors._id: 6565a60e0ff3292512bae26d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6565a60e0ff3292512bae26d/5P6wgEhCI0ndFVgX9KX8k.jpeg
paper_authors.fullname: Mahir Labib Dihan
paper_authors.type: user
paper_authors.user: mahirlabibdihan
paper_authors._id: 6777829e2f92b6f8edf97423
paper_authors.name: Md Tanvir Hassan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T16:25:27.878Z
paper_authors._id: 6334fa3ec3cb9eda932dbff9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1664416305840-noauth.jpeg
paper_authors.fullname: Md. Tanvir Hassan
paper_authors.type: user
paper_authors.user: tanvir14
paper_authors._id: 6777829e2f92b6f8edf97424
paper_authors.name: Md Tanvir Parvez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T16:25:35.522Z
paper_authors._id: 6777b42b6c1219fbace9030a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6777b42b6c1219fbace9030a/zMe5br00qJ5IIV_bdOdKh.jpeg
paper_authors.fullname: Md Tanvir Parvez
paper_authors.type: user
paper_authors.user: tanvirparvez04
paper_authors._id: 6777829e2f92b6f8edf97425
paper_authors.name: Md Hasebul Hasan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-21T10:07:49.011Z
paper_authors._id: 6415fdc458a690df103d99d4
paper_authors.avatarUrl: /avatars/4991d378675ac88912460057b9276851.svg
paper_authors.fullname: Hasebul Hasan Shawon
paper_authors.type: user
paper_authors.user: Hasebul-Hasan
paper_authors._id: 6777829e2f92b6f8edf97426
paper_authors.name: Md Almash Alam
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T16:25:48.276Z
paper_authors._id: 61c1be6bff5ede481bb3e6fa
paper_authors.avatarUrl: /avatars/df4a5611ab183adb41dd4f5190f1f54d.svg
paper_authors.fullname: Md Almash Alam
paper_authors.type: user
paper_authors.user: almash
paper_authors._id: 6777829e2f92b6f8edf97427
paper_authors.name: Muhammad Aamir Cheema
paper_authors._id: 6777829e2f92b6f8edf97428
paper_authors.name: Mohammed Eunus Ali
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-01-03T09:58:41.523Z
paper_authors._id: 6777b4b1414115dd5bcd0991
paper_authors.avatarUrl: /avatars/6d2f6283823c79f70d99732c656d7f4a.svg
paper_authors.fullname: Mohammed Eunus Ali
paper_authors.type: user
paper_authors.user: eunus
paper_authors._id: 6777829e2f92b6f8edf97429
paper_authors.name: Md Rizwan Parvez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T16:26:08.120Z
paper_authors._id: 65ae1c4468139e3c42973fe4
paper_authors.avatarUrl: /avatars/b065a857dd763410caadea37a2dc01c4.svg
paper_authors.fullname: Md Rizwan Parvez
paper_authors.type: user
paper_authors.user: mparvez
paper_title: MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation
    Models
paper_summary: Recent advancements in foundation models have enhanced AI systems'
  capabilities in autonomous tool usage and reasoning. However, their ability in
  location or map-based reasoning - which improves daily life by optimizing
  navigation, facilitating resource discovery, and streamlining logistics - has
  not been systematically studied. To bridge this gap, we introduce MapEval, a
  benchmark designed to assess diverse and complex map-based user queries with
  geo-spatial reasoning. MapEval features three task types (textual, API-based,
  and visual) that require collecting world information via map tools, processing
  heterogeneous geo-spatial contexts (e.g., named entities, travel distances,
  user reviews or ratings, images), and compositional reasoning, which all
  state-of-the-art foundation models find challenging. Comprising 700 unique
  multiple-choice questions about locations across 180 cities and 54 countries,
  MapEval evaluates foundation models' ability to handle spatial relationships,
  map infographics, travel planning, and navigation challenges. Using MapEval, we
  conducted a comprehensive evaluation of 28 prominent foundation models. While
  no single model excelled across all tasks, Claude-3.5-Sonnet, GPT-4o, and
  Gemini-1.5-Pro achieved competitive performance overall. However, substantial
  performance gaps emerged, particularly in MapEval, where agents with
  Claude-3.5-Sonnet outperformed GPT-4o and Gemini-1.5-Pro by 16% and 21%,
  respectively, and the gaps became even more amplified when compared to
  open-source LLMs. Our detailed analyses provide insights into the strengths and
  weaknesses of current models, though all models still fall short of human
  performance by more than 20% on average, struggling with complex map images and
  rigorous geo-spatial reasoning. This gap highlights MapEval's critical role in
  advancing general-purpose foundation models with stronger geo-spatial
  understanding.