title: CityLens: Benchmarking Large Language-Vision Models for Urban
    Socioeconomic Sensing
summary: Understanding urban socioeconomic conditions through visual data is a
  challenging yet essential task for sustainable urban development and policy
  planning. In this work, we introduce CityLens, a comprehensive
  benchmark designed to evaluate the capabilities of large language-vision models
  (LLVMs) in predicting socioeconomic indicators from satellite and street view
  imagery. We construct a multi-modal dataset covering a total of 17 globally
  distributed cities, spanning 6 key domains: economy, education, crime,
  transport, health, and environment, reflecting the multifaceted nature of urban
  life. Based on this dataset, we define 11 prediction tasks and utilize three
  evaluation paradigms: Direct Metric Prediction, Normalized Metric Estimation,
  and Feature-Based Regression. We benchmark 17 state-of-the-art LLVMs across
  these tasks. Our results reveal that while LLVMs demonstrate promising
  perceptual and reasoning capabilities, they still exhibit limitations in
  predicting urban socioeconomic indicators. CityLens provides a unified
  framework for diagnosing these limitations and guiding future efforts in using
  LLVMs to understand and predict urban socioeconomic patterns. Our codes and
  datasets are open-sourced via https://github.com/tsinghua-fib-lab/CityLens.
paper_authors._id: 683ee127345bccde696b3969
paper_authors.name: Tianhui Liu
paper_authors._id: 683ee127345bccde696b396a
paper_authors.name: Jie Feng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T12:33:11.636Z
paper_authors._id: 6465d3bd63e7e09dd02e95c3
paper_authors.avatarUrl: /avatars/b2798bd5f8368f956bf7fab79d9432f0.svg
paper_authors.fullname: Jie Feng
paper_authors.type: user
paper_authors.user: JJ-TMT
paper_authors._id: 683ee127345bccde696b396b
paper_authors.name: Hetian Pang
paper_authors._id: 683ee127345bccde696b396c
paper_authors.name: Xin Zhang
paper_authors._id: 683ee127345bccde696b396d
paper_authors.name: Tianjian Ouyang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T12:43:35.219Z
paper_authors._id: 6566fcb5118497d0af91dc3b
paper_authors.avatarUrl: /avatars/e081d7d1b1657245cd818e5417cdcb2e.svg
paper_authors.fullname: TIANJIAN OUYANG
paper_authors.type: user
paper_authors.user: Ouyangtj
paper_authors._id: 683ee127345bccde696b396e
paper_authors.name: Zhiyuan Zhang
paper_authors._id: 683ee127345bccde696b396f
paper_authors.name: Yong Li
paper_title: CityLens: Benchmarking Large Language-Vision Models for Urban
    Socioeconomic Sensing
paper_summary: Understanding urban socioeconomic conditions through visual data is a
  challenging yet essential task for sustainable urban development and policy
  planning. In this work, we introduce CityLens, a comprehensive
  benchmark designed to evaluate the capabilities of large language-vision models
  (LLVMs) in predicting socioeconomic indicators from satellite and street view
  imagery. We construct a multi-modal dataset covering a total of 17 globally
  distributed cities, spanning 6 key domains: economy, education, crime,
  transport, health, and environment, reflecting the multifaceted nature of urban
  life. Based on this dataset, we define 11 prediction tasks and utilize three
  evaluation paradigms: Direct Metric Prediction, Normalized Metric Estimation,
  and Feature-Based Regression. We benchmark 17 state-of-the-art LLVMs across
  these tasks. Our results reveal that while LLVMs demonstrate promising
  perceptual and reasoning capabilities, they still exhibit limitations in
  predicting urban socioeconomic indicators. CityLens provides a unified
  framework for diagnosing these limitations and guiding future efforts in using
  LLVMs to understand and predict urban socioeconomic patterns. Our codes and
  datasets are open-sourced via https://github.com/tsinghua-fib-lab/CityLens.
paper_ai_summary: CityLens evaluates large language-vision models' capabilities in predicting urban socioeconomic indicators using multi-modal datasets from various global cities.
paper_ai_keywords: CityLens large language-vision models LLVMs satellite imagery street view imagery multi-modal dataset socioeconomic indicators Direct Metric Prediction Normalized Metric Estimation Feature-Based Regression perceptual capabilities reasoning capabilities urban socioeconomic patterns