V-MAGE: A Game Evaluation Framework for Assessing Visual-Centric
  Capabilities in Multimodal Large Language Models
Recent advancements in Multimodal Large Language Models (MLLMs) have led to
significant improvements across various multimodal benchmarks. However, as
evaluations shift from static datasets to open-world, dynamic environments,
current game-based benchmarks remain inadequate because they lack
visual-centric tasks and fail to assess the diverse reasoning skills required
for real-world decision-making. To address this, we introduce Visual-centric
Multiple Abilities Game Evaluation (V-MAGE), a game-based evaluation framework
designed to assess visual reasoning capabilities of MLLMs. V-MAGE features five
diverse games with 30+ handcrafted levels, testing models on core visual skills
such as positioning, trajectory tracking, timing, and visual memory, alongside
higher-level reasoning like long-term planning and deliberation. We use V-MAGE
to evaluate leading MLLMs, revealing significant challenges in their visual
perception and reasoning. In all game environments, the top-performing MLLMs,
as determined by Elo rating comparisons, exhibit a substantial performance gap
compared to humans. Our findings highlight critical limitations, including
various types of perceptual errors made by the models, and suggest potential
avenues for improvement from an agent-centric perspective, such as refining
agent strategies and addressing perceptual inaccuracies. Code is available at
https://github.com/CSU-JPG/V-MAGE.
67f6310fe30d3e5d13a9cbfc
Xiangxi Zheng
claimed_verified
2025-04-09T09:48:14.241Z
673deee2afdcf84dddf74827
/avatars/d2e051ddef816342aa52b98ded109e66.svg
XxZheng
user
Fengx1nn
67f6310fe30d3e5d13a9cbfd
Linjie Li
admin_assigned
2025-04-09T16:43:56.434Z
63db16fff03c3d71ef397206
/avatars/bfb7e0d730b7d03302799d5d2828d97d.svg
Linjie Li
user
linjieli222
67f6310fe30d3e5d13a9cbfe
Zhengyuan Yang
admin_assigned
2025-04-09T16:44:03.328Z
630713411801ecc7d2592a7c
/avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
Zhengyuan Yang
user
zyang39
67f6310fe30d3e5d13a9cbff
Ping Yu
admin_assigned
2025-04-09T16:44:15.037Z
646c004e31968a60a022590e
/avatars/349e2ed4feef1be050fd9903dccec1e1.svg
Ping Yu
user
pingyu
67f6310fe30d3e5d13a9cc00
Alex Jinpeng Wang
admin_assigned
2025-04-09T16:44:21.549Z
62333a88fd7bb4a39b92d387
https://cdn-avatars.huggingface.co/v1/production/uploads/62333a88fd7bb4a39b92d387/e21AhpcXq37Ak_7rZ-Ca9.png
Alex Jinpeng Wang
user
Awiny
67f6310fe30d3e5d13a9cc01
Rui Yan
67f6310fe30d3e5d13a9cc02
Yuan Yao
admin_assigned
2025-04-09T16:44:38.343Z
664618d83a7681520570b58a
/avatars/c03e25ee9bbf3504cdf2959acf81e0fe.svg
Yuan Yao
user
yuanyao
67f6310fe30d3e5d13a9cc03
Lijuan Wang
V-MAGE: A Game Evaluation Framework for Assessing Visual-Centric
  Capabilities in Multimodal Large Language Models
Recent advancements in Multimodal Large Language Models (MLLMs) have led to
significant improvements across various multimodal benchmarks. However, as
evaluations shift from static datasets to open-world, dynamic environments,
current game-based benchmarks remain inadequate because they lack
visual-centric tasks and fail to assess the diverse reasoning skills required
for real-world decision-making. To address this, we introduce Visual-centric
Multiple Abilities Game Evaluation (V-MAGE), a game-based evaluation framework
designed to assess visual reasoning capabilities of MLLMs. V-MAGE features five
diverse games with 30+ handcrafted levels, testing models on core visual skills
such as positioning, trajectory tracking, timing, and visual memory, alongside
higher-level reasoning like long-term planning and deliberation. We use V-MAGE
to evaluate leading MLLMs, revealing significant challenges in their visual
perception and reasoning. In all game environments, the top-performing MLLMs,
as determined by Elo rating comparisons, exhibit a substantial performance gap
compared to humans. Our findings highlight critical limitations, including
various types of perceptual errors made by the models, and suggest potential
avenues for improvement from an agent-centric perspective, such as refining
agent strategies and addressing perceptual inaccuracies. Code is available at
https://github.com/CSU-JPG/V-MAGE.
V-MAGE is a game-based evaluation framework that assesses visual reasoning capabilities of Multimodal Large Language Models (MLLMs) in diverse visual tasks and higher-level reasoning, revealing significant perceptual and reasoning challenges.
Multimodal Large Language Models MLMMs Visual-centric Multiple Abilities Game Evaluation V-MAGE visual reasoning capabilities positioning trajectory tracking timing visual memory long-term planning deliberation Elo rating comparisons