title: V-MAGE: A Game Evaluation Framework for Assessing Visual-Centric
    Capabilities in Multimodal Large Language Models
summary: Recent advancements in Multimodal Large Language Models (MLLMs) have led to
  significant improvements across various multimodal benchmarks. However, as
  evaluations shift from static datasets to open-world, dynamic environments,
  current game-based benchmarks remain inadequate because they lack
  visual-centric tasks and fail to assess the diverse reasoning skills required
  for real-world decision-making. To address this, we introduce Visual-centric
  Multiple Abilities Game Evaluation (V-MAGE), a game-based evaluation framework
  designed to assess visual reasoning capabilities of MLLMs. V-MAGE features five
  diverse games with 30+ handcrafted levels, testing models on core visual skills
  such as positioning, trajectory tracking, timing, and visual memory, alongside
  higher-level reasoning like long-term planning and deliberation. We use V-MAGE
  to evaluate leading MLLMs, revealing significant challenges in their visual
  perception and reasoning. In all game environments, the top-performing MLLMs,
  as determined by Elo rating comparisons, exhibit a substantial performance gap
  compared to humans. Our findings highlight critical limitations, including
  various types of perceptual errors made by the models, and suggest potential
  avenues for improvement from an agent-centric perspective, such as refining
  agent strategies and addressing perceptual inaccuracies. Code is available at
  https://github.com/CSU-JPG/V-MAGE.
paper_authors._id: 67f6310fe30d3e5d13a9cbfc
paper_authors.name: Xiangxi Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-09T09:48:14.241Z
paper_authors._id: 673deee2afdcf84dddf74827
paper_authors.avatarUrl: /avatars/d2e051ddef816342aa52b98ded109e66.svg
paper_authors.fullname: XxZheng
paper_authors.type: user
paper_authors.user: Fengx1nn
paper_authors._id: 67f6310fe30d3e5d13a9cbfd
paper_authors.name: Linjie Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:43:56.434Z
paper_authors._id: 63db16fff03c3d71ef397206
paper_authors.avatarUrl: /avatars/bfb7e0d730b7d03302799d5d2828d97d.svg
paper_authors.fullname: Linjie Li
paper_authors.type: user
paper_authors.user: linjieli222
paper_authors._id: 67f6310fe30d3e5d13a9cbfe
paper_authors.name: Zhengyuan Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:44:03.328Z
paper_authors._id: 630713411801ecc7d2592a7c
paper_authors.avatarUrl: /avatars/fb36f69f03421c3a2a7f72ba0858fa60.svg
paper_authors.fullname: Zhengyuan Yang
paper_authors.type: user
paper_authors.user: zyang39
paper_authors._id: 67f6310fe30d3e5d13a9cbff
paper_authors.name: Ping Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:44:15.037Z
paper_authors._id: 646c004e31968a60a022590e
paper_authors.avatarUrl: /avatars/349e2ed4feef1be050fd9903dccec1e1.svg
paper_authors.fullname: Ping Yu
paper_authors.type: user
paper_authors.user: pingyu
paper_authors._id: 67f6310fe30d3e5d13a9cc00
paper_authors.name: Alex Jinpeng Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:44:21.549Z
paper_authors._id: 62333a88fd7bb4a39b92d387
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62333a88fd7bb4a39b92d387/e21AhpcXq37Ak_7rZ-Ca9.png
paper_authors.fullname: Alex Jinpeng Wang
paper_authors.type: user
paper_authors.user: Awiny
paper_authors._id: 67f6310fe30d3e5d13a9cc01
paper_authors.name: Rui Yan
paper_authors._id: 67f6310fe30d3e5d13a9cc02
paper_authors.name: Yuan Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-09T16:44:38.343Z
paper_authors._id: 664618d83a7681520570b58a
paper_authors.avatarUrl: /avatars/c03e25ee9bbf3504cdf2959acf81e0fe.svg
paper_authors.fullname: Yuan Yao
paper_authors.type: user
paper_authors.user: yuanyao
paper_authors._id: 67f6310fe30d3e5d13a9cc03
paper_authors.name: Lijuan Wang
paper_title: V-MAGE: A Game Evaluation Framework for Assessing Visual-Centric
    Capabilities in Multimodal Large Language Models
paper_summary: Recent advancements in Multimodal Large Language Models (MLLMs) have led to
  significant improvements across various multimodal benchmarks. However, as
  evaluations shift from static datasets to open-world, dynamic environments,
  current game-based benchmarks remain inadequate because they lack
  visual-centric tasks and fail to assess the diverse reasoning skills required
  for real-world decision-making. To address this, we introduce Visual-centric
  Multiple Abilities Game Evaluation (V-MAGE), a game-based evaluation framework
  designed to assess visual reasoning capabilities of MLLMs. V-MAGE features five
  diverse games with 30+ handcrafted levels, testing models on core visual skills
  such as positioning, trajectory tracking, timing, and visual memory, alongside
  higher-level reasoning like long-term planning and deliberation. We use V-MAGE
  to evaluate leading MLLMs, revealing significant challenges in their visual
  perception and reasoning. In all game environments, the top-performing MLLMs,
  as determined by Elo rating comparisons, exhibit a substantial performance gap
  compared to humans. Our findings highlight critical limitations, including
  various types of perceptual errors made by the models, and suggest potential
  avenues for improvement from an agent-centric perspective, such as refining
  agent strategies and addressing perceptual inaccuracies. Code is available at
  https://github.com/CSU-JPG/V-MAGE.
paper_ai_summary: V-MAGE is a game-based evaluation framework that assesses visual reasoning capabilities of Multimodal Large Language Models (MLLMs) in diverse visual tasks and higher-level reasoning, revealing significant perceptual and reasoning challenges.
paper_ai_keywords: Multimodal Large Language Models MLMMs Visual-centric Multiple Abilities Game Evaluation V-MAGE visual reasoning capabilities positioning trajectory tracking timing visual memory long-term planning deliberation Elo rating comparisons