title: ING-VP: MLLMs cannot Play Easy Vision-based Games Yet
summary: As multimodal large language models (MLLMs) continue to demonstrate
  increasingly competitive performance across a broad spectrum of tasks, more
  intricate and comprehensive benchmarks have been developed to assess these
  cutting-edge models. These benchmarks introduce new challenges to core
  capabilities such as perception, reasoning, and planning. However, existing
  multimodal benchmarks fall short in providing a focused evaluation of
  multi-step planning based on spatial relationships in images. To bridge this
  gap, we present ING-VP, the first INteractive Game-based Vision Planning
  benchmark, specifically designed to evaluate the spatial imagination and
  multi-step reasoning abilities of MLLMs. ING-VP features 6 distinct games,
  encompassing 300 levels, each with 6 unique configurations. A single model
  engages in over 60,000 rounds of interaction. The benchmark framework allows
  for multiple comparison settings, including image-text vs. text-only inputs,
  single-step vs. multi-step reasoning, and with-history vs. without-history
  conditions, offering valuable insights into the model's capabilities. We
  evaluated numerous state-of-the-art MLLMs, with the highest-performing model,
  Claude-3.5 Sonnet, achieving an average accuracy of only 3.37%, far below the
  anticipated standard. This work aims to provide a specialized evaluation
  framework to drive advancements in MLLMs' capacity for complex spatial
  reasoning and planning. The code is publicly available at
  https://github.com/Thisisus7/ING-VP.git.
paper_authors._id: 67079d5bdba78ae6a5605e43
paper_authors.name: Haoran Zhang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-10-14T10:18:58.685Z
paper_authors._id: 647c23b6c3c809c69d345c45
paper_authors.avatarUrl: /avatars/b039807814f5a778ca327ead232fc67b.svg
paper_authors.fullname: Haoran Zhang
paper_authors.type: user
paper_authors.user: RickStark
paper_authors._id: 67079d5bdba78ae6a5605e44
paper_authors.name: Hangyu Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:46:00.887Z
paper_authors._id: 646b43deb1202bc77c1024a4
paper_authors.avatarUrl: /avatars/cf791574ab986bac274e7fbcf04e2a59.svg
paper_authors.fullname: hangyu guo
paper_authors.type: user
paper_authors.user: Rosiness
paper_authors._id: 67079d5bdba78ae6a5605e45
paper_authors.name: Shuyue Guo
paper_authors._id: 67079d5bdba78ae6a5605e46
paper_authors.name: Meng Cao
paper_authors._id: 67079d5bdba78ae6a5605e47
paper_authors.name: Wenhao Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-13T20:32:31.028Z
paper_authors._id: 65bb11cb00a03997849e9e85
paper_authors.avatarUrl: /avatars/17022b0254192a837f4fe00d84389cda.svg
paper_authors.fullname: Wenhao Huang
paper_authors.type: user
paper_authors.user: StephenHuang
paper_authors._id: 67079d5bdba78ae6a5605e48
paper_authors.name: Jiaheng Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:45:29.766Z
paper_authors._id: 65377c30e48353201e6fdda0
paper_authors.avatarUrl: /avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
paper_authors.fullname: Jiaheng Liu
paper_authors.type: user
paper_authors.user: CheeryLJH
paper_authors._id: 67079d5bdba78ae6a5605e49
paper_authors.name: Ge Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T09:45:24.241Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_title: ING-VP: MLLMs cannot Play Easy Vision-based Games Yet
paper_summary: As multimodal large language models (MLLMs) continue to demonstrate
  increasingly competitive performance across a broad spectrum of tasks, more
  intricate and comprehensive benchmarks have been developed to assess these
  cutting-edge models. These benchmarks introduce new challenges to core
  capabilities such as perception, reasoning, and planning. However, existing
  multimodal benchmarks fall short in providing a focused evaluation of
  multi-step planning based on spatial relationships in images. To bridge this
  gap, we present ING-VP, the first INteractive Game-based Vision Planning
  benchmark, specifically designed to evaluate the spatial imagination and
  multi-step reasoning abilities of MLLMs. ING-VP features 6 distinct games,
  encompassing 300 levels, each with 6 unique configurations. A single model
  engages in over 60,000 rounds of interaction. The benchmark framework allows
  for multiple comparison settings, including image-text vs. text-only inputs,
  single-step vs. multi-step reasoning, and with-history vs. without-history
  conditions, offering valuable insights into the model's capabilities. We
  evaluated numerous state-of-the-art MLLMs, with the highest-performing model,
  Claude-3.5 Sonnet, achieving an average accuracy of only 3.37%, far below the
  anticipated standard. This work aims to provide a specialized evaluation
  framework to drive advancements in MLLMs' capacity for complex spatial
  reasoning and planning. The code is publicly available at
  https://github.com/Thisisus7/ING-VP.git.
paper_ai_keywords: multimodal large language models (MLLMs) self-supervision diversity textual planning elementary self-supervision benchmarks interactive game-based vision planning ING-VP multi