title: Does Spatial Cognition Emerge in Frontier Models?
summary: Not yet. We present SPACE, a benchmark that systematically evaluates spatial
  cognition in frontier models. Our benchmark builds on decades of research in
  cognitive science. It evaluates large-scale mapping abilities that are brought
  to bear when an organism traverses physical environments, smaller-scale
  reasoning about object shapes and layouts, and cognitive infrastructure such as
  spatial attention and memory. For many tasks, we instantiate parallel
  presentations via text and images, allowing us to benchmark both large language
  models and large multimodal models. Results suggest that contemporary frontier
  models fall short of the spatial intelligence of animals, performing near
  chance level on a number of classic tests of animal cognition.
paper_authors._id: 6707b786ce33247b815f04c4
paper_authors.name: Santhosh Kumar Ramakrishnan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T11:23:47.804Z
paper_authors._id: 653c470c3bd6135805014fb4
paper_authors.avatarUrl: /avatars/3ce1456cafb7dbf6030ea6f775e235cc.svg
paper_authors.fullname: Santhosh Kumar Ramakrishnan
paper_authors.type: user
paper_authors.user: sramakrishnan
paper_authors._id: 6707b786ce33247b815f04c5
paper_authors.name: Erik Wijmans
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T11:23:53.318Z
paper_authors._id: 664ce8c217586a9634370552
paper_authors.avatarUrl: /avatars/b8f96ed0dc321c833d654eab24a2443d.svg
paper_authors.fullname: Erik Wijmans
paper_authors.type: user
paper_authors.user: erikwijmans
paper_authors._id: 6707b786ce33247b815f04c6
paper_authors.name: Philipp Kraehenbuehl
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T11:23:58.952Z
paper_authors._id: 665a93a3212eb0d2f4e0478d
paper_authors.avatarUrl: /avatars/de33167b3dacc518562c62e9122db0fa.svg
paper_authors.fullname: Philipp Kraehenbuehl
paper_authors.type: user
paper_authors.user: philkra
paper_authors._id: 6707b786ce33247b815f04c7
paper_authors.name: Vladlen Koltun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T11:24:04.344Z
paper_authors._id: 6707b1a9356f027ca986d6b1
paper_authors.avatarUrl: /avatars/d32eae81b27dc4f7090196d410e96876.svg
paper_authors.fullname: Vladlen Koltun
paper_authors.type: user
paper_authors.user: vkoltun
paper_title: Does Spatial Cognition Emerge in Frontier Models?
paper_summary: Not yet. We present SPACE, a benchmark that systematically evaluates spatial
  cognition in frontier models. Our benchmark builds on decades of research in
  cognitive science. It evaluates large-scale mapping abilities that are brought
  to bear when an organism traverses physical environments, smaller-scale
  reasoning about object shapes and layouts, and cognitive infrastructure such as
  spatial attention and memory. For many tasks, we instantiate parallel
  presentations via text and images, allowing us to benchmark both large language
  models and large multimodal models. Results suggest that contemporary frontier
  models fall short of the spatial intelligence of animals, performing near
  chance level on a number of classic tests of animal cognition.
paper_ai_keywords: spatial cognition benchmark large-scale mapping object shapes layouts spatial attention spatial memory multimodal models large language models