title: Thinking in Space: How Multimodal Large Language Models See, Remember,
    and Recall Spaces
summary: Humans possess the visual-spatial intelligence to remember spaces from
  sequential visual observations. However, can Multimodal Large Language Models
  (MLLMs) trained on million-scale video datasets also ``think in space'' from
  videos? We present a novel video-based visual-spatial intelligence benchmark
  (VSI-Bench) of over 5,000 question-answer pairs, and find that MLLMs exhibit
  competitive - though subhuman - visual-spatial intelligence. We probe models to
  express how they think in space both linguistically and visually and find that
  while spatial reasoning capabilities remain the primary bottleneck for MLLMs to
  reach higher benchmark performance, local world models and spatial awareness do
  emerge within these models. Notably, prevailing linguistic reasoning techniques
  (e.g., chain-of-thought, self-consistency, tree-of-thoughts) fail to improve
  performance, whereas explicitly generating cognitive maps during
  question-answering enhances MLLMs' spatial distance ability.
paper_authors._id: 676426104fa1553ccc44451f
paper_authors.name: Jihan Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:53:33.378Z
paper_authors._id: 6304baf041387c7f1177a5d2
paper_authors.avatarUrl: /avatars/795c63f2394080eec78ca7981d4a1f78.svg
paper_authors.fullname: Jihan Yang
paper_authors.type: user
paper_authors.user: jihanyang
paper_authors._id: 676426104fa1553ccc444520
paper_authors.name: Shusheng Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-19T18:00:36.526Z
paper_authors._id: 627ccf058b4e56cfc2716425
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1652346592327-noauth.jpeg
paper_authors.fullname: Shusheng Yang
paper_authors.type: user
paper_authors.user: ShushengYang
paper_authors._id: 676426104fa1553ccc444521
paper_authors.name: Anjali W. Gupta
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-21T15:19:58.834Z
paper_authors._id: 654a6c59f8ebcec54510ee56
paper_authors.avatarUrl: /avatars/31089658df2d754ab6e4f6ed2750cc1e.svg
paper_authors.fullname: Anjali W Gupta
paper_authors.type: user
paper_authors.user: anjaliwgupta
paper_authors._id: 676426104fa1553ccc444522
paper_authors.name: Rilyn Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:53:14.391Z
paper_authors._id: 657ad3ed2b365884914652c0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/657ad3ed2b365884914652c0/J86SIaWafJEtqxWH84o0Z.jpeg
paper_authors.fullname: Rilyn Han
paper_authors.type: user
paper_authors.user: rilynhan
paper_authors._id: 676426104fa1553ccc444523
paper_authors.name: Li Fei-Fei
paper_authors._id: 676426104fa1553ccc444524
paper_authors.name: Saining Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:52:57.712Z
paper_authors._id: 6596422646624a86ff3b3bda
paper_authors.avatarUrl: /avatars/216e12b77e45ac5f1fa20932f5745411.svg
paper_authors.fullname: Saining Xie
paper_authors.type: user
paper_authors.user: sainx
paper_title: Thinking in Space: How Multimodal Large Language Models See, Remember,
    and Recall Spaces
paper_summary: Humans possess the visual-spatial intelligence to remember spaces from
  sequential visual observations. However, can Multimodal Large Language Models
  (MLLMs) trained on million-scale video datasets also ``think in space'' from
  videos? We present a novel video-based visual-spatial intelligence benchmark
  (VSI-Bench) of over 5,000 question-answer pairs, and find that MLLMs exhibit
  competitive - though subhuman - visual-spatial intelligence. We probe models to
  express how they think in space both linguistically and visually and find that
  while spatial reasoning capabilities remain the primary bottleneck for MLLMs to
  reach higher benchmark performance, local world models and spatial awareness do
  emerge within these models. Notably, prevailing linguistic reasoning techniques
  (e.g., chain-of-thought, self-consistency, tree-of-thoughts) fail to improve
  performance, whereas explicitly generating cognitive maps during
  question-answering enhances MLLMs' spatial distance ability.
paper_ai_keywords: Multimodal Large Language Models (MLLMs) video-based visual-spatial intelligence benchmark (VSI-Bench) spatial reasoning capabilities local world models spatial awareness chain-of-thought self-consistency tree-of-thoughts cognitive maps spatial distance ability