Evaluating Tool-Augmented Agents in Remote Sensing Platforms
<|reference_start|>Evaluating Tool-Augmented Agents in Remote Sensing Platforms: Tool-augmented Large Language Models (LLMs) have shown impressive capabilities in remote sensing (RS) applications. However, existing benchmarks assume question-answering input templates over predefined image-text data pairs. These standalone instructions neglect the intricacies of realistic user-grounded tasks. Consider a geospatial analyst: they zoom in a map area, they draw a region over which to collect satellite imagery, and they succinctly ask "Detect all objects here". Where is `here`, if it is not explicitly hardcoded in the image-text template, but instead is implied by the system state, e.g., the live map positioning? To bridge this gap, we present GeoLLM-QA, a benchmark designed to capture long sequences of verbal, visual, and click-based actions on a real UI platform. Through in-depth evaluation of state-of-the-art LLMs over a diverse set of 1,000 tasks, we offer insights towards stronger agents for RS applications.<|reference_end|>
@article{singh2024evaluating,
    title={Evaluating Tool-Augmented Agents in Remote Sensing Platforms},
    author={Simranjit Singh, Michael Fore, Dimitrios Stamoulis},
    journal={arXiv preprint arXiv:2405.00709},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.00709},
    primaryClass={cs.CL cs.AI cs.LG}
}