EarthMind: Towards Multi-Granular and Multi-Sensor Earth Observation
  with Large Multimodal Models
Large Multimodal Models (LMMs) have demonstrated strong performance in
various vision-language tasks. However, they often struggle to comprehensively
understand Earth Observation (EO) data, which is critical for monitoring the
environment and the effects of human activity on it. In this work, we present
EarthMind, a novel vision-language framework for multi-granular and
multi-sensor EO data understanding. EarthMind features two core components: (1)
Spatial Attention Prompting (SAP), which reallocates attention within the LLM
to enhance pixel-level understanding; and (2) Cross-modal Fusion, which aligns
heterogeneous modalities into a shared space and adaptively reweighs tokens
based on their information density for effective fusion. To facilitate
multi-sensor fusion evaluation, we propose EarthMind-Bench, a comprehensive
benchmark with over 2,000 human-annotated multi-sensor image-question pairs,
covering a wide range of perception and reasoning tasks. Extensive experiments
demonstrate the effectiveness of EarthMind. It achieves state-of-the-art
performance on EarthMind-Bench, surpassing GPT-4o despite being only 4B in
scale. Moreover, EarthMind outperforms existing methods on multiple public EO
benchmarks, showcasing its potential to handle both multi-granular and
multi-sensor challenges in a unified framework.
683eaa8ed8d42fc832445ebd
Yan Shu
claimed_verified
2025-06-03T12:33:48.379Z
65c4f99b27736b5b86c2cbda
/avatars/8789b231ec16073ea0229c28f1f1dd06.svg
Yan Shu
user
sy1998
683eaa8ed8d42fc832445ebe
Bin Ren
683eaa8ed8d42fc832445ebf
Zhitong Xiong
claimed_verified
2025-07-01T10:14:55.618Z
63931121f4be79ef705f6e18
/avatars/75fc223936df47c7ace46999bc05c30f.svg
zhitong
user
earthflow
683eaa8ed8d42fc832445ec0
Danda Pani Paudel
683eaa8ed8d42fc832445ec1
Luc Van Gool
683eaa8ed8d42fc832445ec2
Begum Demir
683eaa8ed8d42fc832445ec3
Nicu Sebe
683eaa8ed8d42fc832445ec4
Paolo Rota
admin_assigned
2025-06-03T12:40:42.097Z
62f3b1ea81861bd9bc5c5538
/avatars/0aef9ac5bfa91b9894166fe3c29925da.svg
Paolo Rota
user
paolorota
EarthMind: Towards Multi-Granular and Multi-Sensor Earth Observation
  with Large Multimodal Models
Large Multimodal Models (LMMs) have demonstrated strong performance in
various vision-language tasks. However, they often struggle to comprehensively
understand Earth Observation (EO) data, which is critical for monitoring the
environment and the effects of human activity on it. In this work, we present
EarthMind, a novel vision-language framework for multi-granular and
multi-sensor EO data understanding. EarthMind features two core components: (1)
Spatial Attention Prompting (SAP), which reallocates attention within the LLM
to enhance pixel-level understanding; and (2) Cross-modal Fusion, which aligns
heterogeneous modalities into a shared space and adaptively reweighs tokens
based on their information density for effective fusion. To facilitate
multi-sensor fusion evaluation, we propose EarthMind-Bench, a comprehensive
benchmark with over 2,000 human-annotated multi-sensor image-question pairs,
covering a wide range of perception and reasoning tasks. Extensive experiments
demonstrate the effectiveness of EarthMind. It achieves state-of-the-art
performance on EarthMind-Bench, surpassing GPT-4o despite being only 4B in
scale. Moreover, EarthMind outperforms existing methods on multiple public EO
benchmarks, showcasing its potential to handle both multi-granular and
multi-sensor challenges in a unified framework.
EarthMind is a vision-language framework that uses spatial attention prompting and cross-modal fusion for efficient multi-granular and multi-sensor Earth Observation data understanding, outperforming larger models on specialized benchmarks.
Spatial Attention Prompting Cross-modal Fusion Earth Observation multi-granular multi-sensor EarthMind-Bench