PLUGH: A Benchmark for Spatial Understanding and Reasoning in Large Language Models
<|reference_start|>PLUGH: A Benchmark for Spatial Understanding and Reasoning in Large Language Models: We present PLUGH (https://www.urbandictionary.com/define.php?term=plugh), a modern benchmark that currently consists of 5 tasks, each with 125 input texts extracted from 48 different games and representing 61 different (non-isomorphic) spatial graphs to assess the abilities of Large Language Models (LLMs) for spatial understanding and reasoning. Our evaluation of API-based and open-sourced LLMs shows that while some commercial LLMs exhibit strong reasoning abilities, open-sourced competitors can demonstrate almost the same level of quality; however, all models still have significant room for improvement. We identify typical reasons for LLM failures and discuss possible ways to deal with them. Datasets and evaluation code are released (https://github.com/altsoph/PLUGH).<|reference_end|>
@article{tikhonov2024plugh:,
    title={PLUGH: A Benchmark for Spatial Understanding and Reasoning in Large
  Language Models},
    author={Alexey Tikhonov},
    journal={arXiv preprint arXiv:2408.04648},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.04648},
    primaryClass={cs.CL cs.AI cs.IR}
}