UrBench: A Comprehensive Benchmark for Evaluating Large Multimodal
  Models in Multi-View Urban Scenarios
Recent evaluations of Large Multimodal Models (LMMs) have explored their
capabilities in various domains, with only few benchmarks specifically focusing
on urban environments. Moreover, existing urban benchmarks have been limited to
evaluating LMMs with basic region-level urban tasks under singular views,
leading to incomplete evaluations of LMMs' abilities in urban environments. To
address these issues, we present UrBench, a comprehensive benchmark designed
for evaluating LMMs in complex multi-view urban scenarios. UrBench contains
11.6K meticulously curated questions at both region-level and role-level that
cover 4 task dimensions: Geo-Localization, Scene Reasoning, Scene
Understanding, and Object Understanding, totaling 14 task types. In
constructing UrBench, we utilize data from existing datasets and additionally
collect data from 11 cities, creating new annotations using a cross-view
detection-matching method. With these images and annotations, we then integrate
LMM-based, rule-based, and human-based methods to construct large-scale
high-quality questions. Our evaluations on 21 LMMs show that current LMMs
struggle in the urban environments in several aspects. Even the best performing
GPT-4o lags behind humans in most tasks, ranging from simple tasks such as
counting to complex tasks such as orientation, localization and object
attribute recognition, with an average performance gap of 17.4%. Our benchmark
also reveals that LMMs exhibit inconsistent behaviors with different urban
views, especially with respect to understanding cross-view relations. UrBench
datasets and benchmark results will be publicly available at
https://opendatalab.github.io/UrBench/.
66d59d45d97d73e4b6019c3a
Baichuan Zhou
claimed_verified
2024-09-03T07:48:32.619Z
646325085897b675c65aea0f
/avatars/28ce7388f9318b49bdd0a5594c0f6732.svg
Baichuan Zhou
user
bczhou
66d59d45d97d73e4b6019c3b
Haote Yang
claimed_verified
2024-09-03T14:49:01.251Z
65fd45473ccf43503350d837
https://cdn-avatars.huggingface.co/v1/production/uploads/65fd45473ccf43503350d837/i8r1uUpD0dP1vrrWucO6E.jpeg
Haote Yang
user
Hoter
66d59d45d97d73e4b6019c3c
Dairong Chen
66d59d45d97d73e4b6019c3d
Junyan Ye
66d59d45d97d73e4b6019c3e
Tianyi Bai
claimed_verified
2024-10-14T07:48:56.579Z
641802c31f1f3b0fa812225c
https://cdn-avatars.huggingface.co/v1/production/uploads/641802c31f1f3b0fa812225c/i-OFWm8d37ofsmuvvAKYR.png
becca_bai
user
beccabai
66d59d45d97d73e4b6019c3f
Jinhua Yu
66d59d45d97d73e4b6019c40
Songyang Zhang
claimed_verified
2024-09-25T07:35:13.988Z
630716d11801ecc7d2595021
/avatars/2d36a880ce4a3cf7efc5ff3987dbeaf3.svg
Songyang Zhang
user
zsytony
66d59d45d97d73e4b6019c41
Dahua Lin
66d59d45d97d73e4b6019c42
Conghui He
66d59d45d97d73e4b6019c43
Weijia Li
claimed_verified
2024-09-03T07:48:29.653Z
66d5b56c77a026c3d2086a79
/avatars/45da07fd82fd455955faa05b27a6393f.svg
Weijia Li
user
liweijia
UrBench: A Comprehensive Benchmark for Evaluating Large Multimodal
  Models in Multi-View Urban Scenarios
Recent evaluations of Large Multimodal Models (LMMs) have explored their
capabilities in various domains, with only few benchmarks specifically focusing
on urban environments. Moreover, existing urban benchmarks have been limited to
evaluating LMMs with basic region-level urban tasks under singular views,
leading to incomplete evaluations of LMMs' abilities in urban environments. To
address these issues, we present UrBench, a comprehensive benchmark designed
for evaluating LMMs in complex multi-view urban scenarios. UrBench contains
11.6K meticulously curated questions at both region-level and role-level that
cover 4 task dimensions: Geo-Localization, Scene Reasoning, Scene
Understanding, and Object Understanding, totaling 14 task types. In
constructing UrBench, we utilize data from existing datasets and additionally
collect data from 11 cities, creating new annotations using a cross-view
detection-matching method. With these images and annotations, we then integrate
LMM-based, rule-based, and human-based methods to construct large-scale
high-quality questions. Our evaluations on 21 LMMs show that current LMMs
struggle in the urban environments in several aspects. Even the best performing
GPT-4o lags behind humans in most tasks, ranging from simple tasks such as
counting to complex tasks such as orientation, localization and object
attribute recognition, with an average performance gap of 17.4%. Our benchmark
also reveals that LMMs exhibit inconsistent behaviors with different urban
views, especially with respect to understanding cross-view relations. UrBench
datasets and benchmark results will be publicly available at
https://opendatalab.github.io/UrBench/.
UrBench is a comprehensive benchmark for evaluating Large Multimodal Models in complex, multi-view urban scenarios, revealing gaps in model performance compared to humans.
Large Multimodal Models UrBench urban environments region-level role-level Geo-Localization Scene Reasoning Scene Understanding Object Understanding cross-view detection-matching LMM-based rule-based human-based