title: EarthMind: Towards Multi-Granular and Multi-Sensor Earth Observation
    with Large Multimodal Models
summary: Large Multimodal Models (LMMs) have demonstrated strong performance in
  various vision-language tasks. However, they often struggle to comprehensively
  understand Earth Observation (EO) data, which is critical for monitoring the
  environment and the effects of human activity on it. In this work, we present
  EarthMind, a novel vision-language framework for multi-granular and
  multi-sensor EO data understanding. EarthMind features two core components: (1)
  Spatial Attention Prompting (SAP), which reallocates attention within the LLM
  to enhance pixel-level understanding; and (2) Cross-modal Fusion, which aligns
  heterogeneous modalities into a shared space and adaptively reweighs tokens
  based on their information density for effective fusion. To facilitate
  multi-sensor fusion evaluation, we propose EarthMind-Bench, a comprehensive
  benchmark with over 2,000 human-annotated multi-sensor image-question pairs,
  covering a wide range of perception and reasoning tasks. Extensive experiments
  demonstrate the effectiveness of EarthMind. It achieves state-of-the-art
  performance on EarthMind-Bench, surpassing GPT-4o despite being only 4B in
  scale. Moreover, EarthMind outperforms existing methods on multiple public EO
  benchmarks, showcasing its potential to handle both multi-granular and
  multi-sensor challenges in a unified framework.
paper_authors._id: 683eaa8ed8d42fc832445ebd
paper_authors.name: Yan Shu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T12:33:48.379Z
paper_authors._id: 65c4f99b27736b5b86c2cbda
paper_authors.avatarUrl: /avatars/8789b231ec16073ea0229c28f1f1dd06.svg
paper_authors.fullname: Yan Shu
paper_authors.type: user
paper_authors.user: sy1998
paper_authors._id: 683eaa8ed8d42fc832445ebe
paper_authors.name: Bin Ren
paper_authors._id: 683eaa8ed8d42fc832445ebf
paper_authors.name: Zhitong Xiong
paper_authors._id: 683eaa8ed8d42fc832445ec0
paper_authors.name: Danda Pani Paudel
paper_authors._id: 683eaa8ed8d42fc832445ec1
paper_authors.name: Luc Van Gool
paper_authors._id: 683eaa8ed8d42fc832445ec2
paper_authors.name: Begum Demir
paper_authors._id: 683eaa8ed8d42fc832445ec3
paper_authors.name: Nicu Sebe
paper_authors._id: 683eaa8ed8d42fc832445ec4
paper_authors.name: Paolo Rota
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-03T12:40:42.097Z
paper_authors._id: 62f3b1ea81861bd9bc5c5538
paper_authors.avatarUrl: /avatars/0aef9ac5bfa91b9894166fe3c29925da.svg
paper_authors.fullname: Paolo Rota
paper_authors.type: user
paper_authors.user: paolorota
paper_title: EarthMind: Towards Multi-Granular and Multi-Sensor Earth Observation
    with Large Multimodal Models
paper_summary: Large Multimodal Models (LMMs) have demonstrated strong performance in
  various vision-language tasks. However, they often struggle to comprehensively
  understand Earth Observation (EO) data, which is critical for monitoring the
  environment and the effects of human activity on it. In this work, we present
  EarthMind, a novel vision-language framework for multi-granular and
  multi-sensor EO data understanding. EarthMind features two core components: (1)
  Spatial Attention Prompting (SAP), which reallocates attention within the LLM
  to enhance pixel-level understanding; and (2) Cross-modal Fusion, which aligns
  heterogeneous modalities into a shared space and adaptively reweighs tokens
  based on their information density for effective fusion. To facilitate
  multi-sensor fusion evaluation, we propose EarthMind-Bench, a comprehensive
  benchmark with over 2,000 human-annotated multi-sensor image-question pairs,
  covering a wide range of perception and reasoning tasks. Extensive experiments
  demonstrate the effectiveness of EarthMind. It achieves state-of-the-art
  performance on EarthMind-Bench, surpassing GPT-4o despite being only 4B in
  scale. Moreover, EarthMind outperforms existing methods on multiple public EO
  benchmarks, showcasing its potential to handle both multi-granular and
  multi-sensor challenges in a unified framework.
paper_ai_summary: EarthMind is a vision-language framework that uses spatial attention prompting and cross-modal fusion for efficient multi-granular and multi-sensor Earth Observation data understanding, outperforming larger models on specialized benchmarks.
paper_ai_keywords: Spatial Attention Prompting Cross-modal Fusion Earth Observation multi-granular multi-sensor EarthMind-Bench