MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation
  Models
Recent advancements in foundation models have enhanced AI systems'
capabilities in autonomous tool usage and reasoning. However, their ability in
location or map-based reasoning - which improves daily life by optimizing
navigation, facilitating resource discovery, and streamlining logistics - has
not been systematically studied. To bridge this gap, we introduce MapEval, a
benchmark designed to assess diverse and complex map-based user queries with
geo-spatial reasoning. MapEval features three task types (textual, API-based,
and visual) that require collecting world information via map tools, processing
heterogeneous geo-spatial contexts (e.g., named entities, travel distances,
user reviews or ratings, images), and compositional reasoning, which all
state-of-the-art foundation models find challenging. Comprising 700 unique
multiple-choice questions about locations across 180 cities and 54 countries,
MapEval evaluates foundation models' ability to handle spatial relationships,
map infographics, travel planning, and navigation challenges. Using MapEval, we
conducted a comprehensive evaluation of 28 prominent foundation models. While
no single model excelled across all tasks, Claude-3.5-Sonnet, GPT-4o, and
Gemini-1.5-Pro achieved competitive performance overall. However, substantial
performance gaps emerged, particularly in MapEval, where agents with
Claude-3.5-Sonnet outperformed GPT-4o and Gemini-1.5-Pro by 16% and 21%,
respectively, and the gaps became even more amplified when compared to
open-source LLMs. Our detailed analyses provide insights into the strengths and
weaknesses of current models, though all models still fall short of human
performance by more than 20% on average, struggling with complex map images and
rigorous geo-spatial reasoning. This gap highlights MapEval's critical role in
advancing general-purpose foundation models with stronger geo-spatial
understanding.
6777829e2f92b6f8edf97422
Mahir Labib Dihan
claimed_verified
2025-01-03T14:03:04.107Z
6565a60e0ff3292512bae26d
https://cdn-avatars.huggingface.co/v1/production/uploads/6565a60e0ff3292512bae26d/5P6wgEhCI0ndFVgX9KX8k.jpeg
Mahir Labib Dihan
user
mahirlabibdihan
6777829e2f92b6f8edf97423
Md Tanvir Hassan
admin_assigned
2025-01-06T16:25:27.878Z
6334fa3ec3cb9eda932dbff9
https://cdn-avatars.huggingface.co/v1/production/uploads/1664416305840-noauth.jpeg
Md. Tanvir Hassan
user
tanvir14
6777829e2f92b6f8edf97424
Md Tanvir Parvez
admin_assigned
2025-01-06T16:25:35.522Z
6777b42b6c1219fbace9030a
https://cdn-avatars.huggingface.co/v1/production/uploads/6777b42b6c1219fbace9030a/zMe5br00qJ5IIV_bdOdKh.jpeg
Md Tanvir Parvez
user
tanvirparvez04
6777829e2f92b6f8edf97425
Md Hasebul Hasan
claimed_verified
2025-01-21T10:07:49.011Z
6415fdc458a690df103d99d4
/avatars/4991d378675ac88912460057b9276851.svg
Hasebul Hasan Shawon
user
Hasebul-Hasan
6777829e2f92b6f8edf97426
Md Almash Alam
admin_assigned
2025-01-06T16:25:48.276Z
61c1be6bff5ede481bb3e6fa
/avatars/df4a5611ab183adb41dd4f5190f1f54d.svg
Md Almash Alam
user
almash
6777829e2f92b6f8edf97427
Muhammad Aamir Cheema
6777829e2f92b6f8edf97428
Mohammed Eunus Ali
extracted_pending
2025-01-03T09:58:41.523Z
6777b4b1414115dd5bcd0991
/avatars/6d2f6283823c79f70d99732c656d7f4a.svg
Mohammed Eunus Ali
user
eunus
6777829e2f92b6f8edf97429
Md Rizwan Parvez
admin_assigned
2025-01-06T16:26:08.120Z
65ae1c4468139e3c42973fe4
/avatars/b065a857dd763410caadea37a2dc01c4.svg
Md Rizwan Parvez
user
mparvez
MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation
  Models
Recent advancements in foundation models have enhanced AI systems'
capabilities in autonomous tool usage and reasoning. However, their ability in
location or map-based reasoning - which improves daily life by optimizing
navigation, facilitating resource discovery, and streamlining logistics - has
not been systematically studied. To bridge this gap, we introduce MapEval, a
benchmark designed to assess diverse and complex map-based user queries with
geo-spatial reasoning. MapEval features three task types (textual, API-based,
and visual) that require collecting world information via map tools, processing
heterogeneous geo-spatial contexts (e.g., named entities, travel distances,
user reviews or ratings, images), and compositional reasoning, which all
state-of-the-art foundation models find challenging. Comprising 700 unique
multiple-choice questions about locations across 180 cities and 54 countries,
MapEval evaluates foundation models' ability to handle spatial relationships,
map infographics, travel planning, and navigation challenges. Using MapEval, we
conducted a comprehensive evaluation of 28 prominent foundation models. While
no single model excelled across all tasks, Claude-3.5-Sonnet, GPT-4o, and
Gemini-1.5-Pro achieved competitive performance overall. However, substantial
performance gaps emerged, particularly in MapEval, where agents with
Claude-3.5-Sonnet outperformed GPT-4o and Gemini-1.5-Pro by 16% and 21%,
respectively, and the gaps became even more amplified when compared to
open-source LLMs. Our detailed analyses provide insights into the strengths and
weaknesses of current models, though all models still fall short of human
performance by more than 20% on average, struggling with complex map images and
rigorous geo-spatial reasoning. This gap highlights MapEval's critical role in
advancing general-purpose foundation models with stronger geo-spatial
understanding.
MapEval assesses foundation models' geo-spatial reasoning and map-based query handling, revealing performance gaps and suggesting improvements in spatial understanding.
geo-spatial reasoning benchmark map-based queries textual API-based visual multiple-choice questions spatial relationships map infographics travel planning navigation challenges foundation models