title: UrBench: A Comprehensive Benchmark for Evaluating Large Multimodal
    Models in Multi-View Urban Scenarios
summary: Recent evaluations of Large Multimodal Models (LMMs) have explored their
  capabilities in various domains, with only few benchmarks specifically focusing
  on urban environments. Moreover, existing urban benchmarks have been limited to
  evaluating LMMs with basic region-level urban tasks under singular views,
  leading to incomplete evaluations of LMMs' abilities in urban environments. To
  address these issues, we present UrBench, a comprehensive benchmark designed
  for evaluating LMMs in complex multi-view urban scenarios. UrBench contains
  11.6K meticulously curated questions at both region-level and role-level that
  cover 4 task dimensions: Geo-Localization, Scene Reasoning, Scene
  Understanding, and Object Understanding, totaling 14 task types. In
  constructing UrBench, we utilize data from existing datasets and additionally
  collect data from 11 cities, creating new annotations using a cross-view
  detection-matching method. With these images and annotations, we then integrate
  LMM-based, rule-based, and human-based methods to construct large-scale
  high-quality questions. Our evaluations on 21 LMMs show that current LMMs
  struggle in the urban environments in several aspects. Even the best performing
  GPT-4o lags behind humans in most tasks, ranging from simple tasks such as
  counting to complex tasks such as orientation, localization and object
  attribute recognition, with an average performance gap of 17.4%. Our benchmark
  also reveals that LMMs exhibit inconsistent behaviors with different urban
  views, especially with respect to understanding cross-view relations. UrBench
  datasets and benchmark results will be publicly available at
  https://opendatalab.github.io/UrBench/.
paper_authors._id: 66d59d45d97d73e4b6019c3a
paper_authors.name: Baichuan Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-03T07:48:32.619Z
paper_authors._id: 646325085897b675c65aea0f
paper_authors.avatarUrl: /avatars/28ce7388f9318b49bdd0a5594c0f6732.svg
paper_authors.fullname: Baichuan Zhou
paper_authors.type: user
paper_authors.user: bczhou
paper_authors._id: 66d59d45d97d73e4b6019c3b
paper_authors.name: Haote Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-03T14:49:01.251Z
paper_authors._id: 65fd45473ccf43503350d837
paper_authors.avatarUrl: /avatars/11b9679945b3c89b142f0d62a312f362.svg
paper_authors.fullname: Haote Yang
paper_authors.type: user
paper_authors.user: Hoter
paper_authors._id: 66d59d45d97d73e4b6019c3c
paper_authors.name: Dairong Chen
paper_authors._id: 66d59d45d97d73e4b6019c3d
paper_authors.name: Junyan Ye
paper_authors._id: 66d59d45d97d73e4b6019c3e
paper_authors.name: Tianyi Bai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T07:48:56.579Z
paper_authors._id: 641802c31f1f3b0fa812225c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641802c31f1f3b0fa812225c/i-OFWm8d37ofsmuvvAKYR.png
paper_authors.fullname: becca_bai
paper_authors.type: user
paper_authors.user: beccabai
paper_authors._id: 66d59d45d97d73e4b6019c3f
paper_authors.name: Jinhua Yu
paper_authors._id: 66d59d45d97d73e4b6019c40
paper_authors.name: Songyang Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-25T07:35:13.988Z
paper_authors._id: 630716d11801ecc7d2595021
paper_authors.avatarUrl: /avatars/2d36a880ce4a3cf7efc5ff3987dbeaf3.svg
paper_authors.fullname: Songyang Zhang
paper_authors.type: user
paper_authors.user: zsytony
paper_authors._id: 66d59d45d97d73e4b6019c41
paper_authors.name: Dahua Lin
paper_authors._id: 66d59d45d97d73e4b6019c42
paper_authors.name: Conghui He
paper_authors._id: 66d59d45d97d73e4b6019c43
paper_authors.name: Weijia Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-03T07:48:29.653Z
paper_authors._id: 66d5b56c77a026c3d2086a79
paper_authors.avatarUrl: /avatars/45da07fd82fd455955faa05b27a6393f.svg
paper_authors.fullname: Weijia Li
paper_authors.type: user
paper_authors.user: liweijia
paper_title: UrBench: A Comprehensive Benchmark for Evaluating Large Multimodal
    Models in Multi-View Urban Scenarios
paper_summary: Recent evaluations of Large Multimodal Models (LMMs) have explored their
  capabilities in various domains, with only few benchmarks specifically focusing
  on urban environments. Moreover, existing urban benchmarks have been limited to
  evaluating LMMs with basic region-level urban tasks under singular views,
  leading to incomplete evaluations of LMMs' abilities in urban environments. To
  address these issues, we present UrBench, a comprehensive benchmark designed
  for evaluating LMMs in complex multi-view urban scenarios. UrBench contains
  11.6K meticulously curated questions at both region-level and role-level that
  cover 4 task dimensions: Geo-Localization, Scene Reasoning, Scene
  Understanding, and Object Understanding, totaling 14 task types. In
  constructing UrBench, we utilize data from existing datasets and additionally
  collect data from 11 cities, creating new annotations using a cross-view
  detection-matching method. With these images and annotations, we then integrate
  LMM-based, rule-based, and human-based methods to construct large-scale
  high-quality questions. Our evaluations on 21 LMMs show that current LMMs
  struggle in the urban environments in several aspects. Even the best performing
  GPT-4o lags behind humans in most tasks, ranging from simple tasks such as
  counting to complex tasks such as orientation, localization and object
  attribute recognition, with an average performance gap of 17.4%. Our benchmark
  also reveals that LMMs exhibit inconsistent behaviors with different urban
  views, especially with respect to understanding cross-view relations. UrBench
  datasets and benchmark results will be publicly available at
  https://opendatalab.github.io/UrBench/.
paper_ai_keywords: Geo-Localization Scene Reasoning Scene Understanding Object Understanding cross-view detection-matching method LMM-based rule-based human-based orientation localization object attribute recognition