ING-VP: MLLMs cannot Play Easy Vision-based Games Yet
As multimodal large language models (MLLMs) continue to demonstrate
increasingly competitive performance across a broad spectrum of tasks, more
intricate and comprehensive benchmarks have been developed to assess these
cutting-edge models. These benchmarks introduce new challenges to core
capabilities such as perception, reasoning, and planning. However, existing
multimodal benchmarks fall short in providing a focused evaluation of
multi-step planning based on spatial relationships in images. To bridge this
gap, we present ING-VP, the first INteractive Game-based Vision Planning
benchmark, specifically designed to evaluate the spatial imagination and
multi-step reasoning abilities of MLLMs. ING-VP features 6 distinct games,
encompassing 300 levels, each with 6 unique configurations. A single model
engages in over 60,000 rounds of interaction. The benchmark framework allows
for multiple comparison settings, including image-text vs. text-only inputs,
single-step vs. multi-step reasoning, and with-history vs. without-history
conditions, offering valuable insights into the model's capabilities. We
evaluated numerous state-of-the-art MLLMs, with the highest-performing model,
Claude-3.5 Sonnet, achieving an average accuracy of only 3.37%, far below the
anticipated standard. This work aims to provide a specialized evaluation
framework to drive advancements in MLLMs' capacity for complex spatial
reasoning and planning. The code is publicly available at
https://github.com/Thisisus7/ING-VP.git.
67079d5bdba78ae6a5605e43
Haoran Zhang
extracted_pending
2024-10-14T10:18:58.685Z
647c23b6c3c809c69d345c45
/avatars/b039807814f5a778ca327ead232fc67b.svg
Haoran Zhang
user
RickStark
67079d5bdba78ae6a5605e44
Hangyu Guo
admin_assigned
2024-10-10T09:46:00.887Z
646b43deb1202bc77c1024a4
/avatars/cf791574ab986bac274e7fbcf04e2a59.svg
hangyu guo
user
Rosiness
67079d5bdba78ae6a5605e45
Shuyue Guo
67079d5bdba78ae6a5605e46
Meng Cao
67079d5bdba78ae6a5605e47
Wenhao Huang
claimed_verified
2024-10-13T20:32:31.028Z
65bb11cb00a03997849e9e85
/avatars/17022b0254192a837f4fe00d84389cda.svg
Wenhao Huang
user
StephenHuang
67079d5bdba78ae6a5605e48
Jiaheng Liu
admin_assigned
2024-10-10T09:45:29.766Z
65377c30e48353201e6fdda0
/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
Jiaheng Liu
user
CheeryLJH
67079d5bdba78ae6a5605e49
Ge Zhang
admin_assigned
2024-10-10T09:45:24.241Z
638efcf4c67af472d316d424
/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
Ge Zhang
user
zhangysk
ING-VP: MLLMs cannot Play Easy Vision-based Games Yet
As multimodal large language models (MLLMs) continue to demonstrate
increasingly competitive performance across a broad spectrum of tasks, more
intricate and comprehensive benchmarks have been developed to assess these
cutting-edge models. These benchmarks introduce new challenges to core
capabilities such as perception, reasoning, and planning. However, existing
multimodal benchmarks fall short in providing a focused evaluation of
multi-step planning based on spatial relationships in images. To bridge this
gap, we present ING-VP, the first INteractive Game-based Vision Planning
benchmark, specifically designed to evaluate the spatial imagination and
multi-step reasoning abilities of MLLMs. ING-VP features 6 distinct games,
encompassing 300 levels, each with 6 unique configurations. A single model
engages in over 60,000 rounds of interaction. The benchmark framework allows
for multiple comparison settings, including image-text vs. text-only inputs,
single-step vs. multi-step reasoning, and with-history vs. without-history
conditions, offering valuable insights into the model's capabilities. We
evaluated numerous state-of-the-art MLLMs, with the highest-performing model,
Claude-3.5 Sonnet, achieving an average accuracy of only 3.37%, far below the
anticipated standard. This work aims to provide a specialized evaluation
framework to drive advancements in MLLMs' capacity for complex spatial
reasoning and planning. The code is publicly available at
https://github.com/Thisisus7/ING-VP.git.
ING-VP is a new benchmark designed to evaluate multimodal large language models' spatial imagination and multi-step reasoning through interactive game-based vision planning tasks.
multimodal large language models perception reasoning planning spatial relationships interactive game-based vision planning spatial imagination multi-step reasoning image-text inputs single-step reasoning multi-step reasoning with-history without-history accuracy