title: VideoGameBench: Can Vision-Language Models complete popular video games?
summary: Vision-language models (VLMs) have achieved strong results on coding and math
  benchmarks that are challenging for humans, yet their ability to perform tasks
  that come naturally to humans--such as perception, spatial navigation, and
  memory management--remains understudied. Real video games are crafted to be
  intuitive for humans to learn and master by leveraging innate inductive biases,
  making them an ideal testbed for evaluating such capabilities in VLMs. To this
  end, we introduce VideoGameBench, a benchmark consisting of 10 popular video
  games from the 1990s that VLMs directly interact with in real-time.
  VideoGameBench challenges models to complete entire games with access to only
  raw visual inputs and a high-level description of objectives and controls, a
  significant departure from existing setups that rely on game-specific
  scaffolding and auxiliary information. We keep three of the games secret to
  encourage solutions that generalize to unseen environments. Our experiments
  show that frontier vision-language models struggle to progress beyond the
  beginning of each game. We find inference latency to be a major limitation of
  frontier models in the real-time setting; therefore, we introduce
  VideoGameBench Lite, a setting where the game pauses while waiting for the LM's
  next action. The best performing model, Gemini 2.5 Pro, completes only 0.48% of
  VideoGameBench and 1.6% of VideoGameBench Lite. We hope that the formalization
  of the human skills mentioned above into this benchmark motivates progress in
  these research directions.
paper_authors._id: 6837123c7b41ef0965ce8f44
paper_authors.name: Alex L. Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-30T06:54:56.688Z
paper_authors._id: 67bcaba1608ec2bdb922e8ec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/y08F7lviS-83uDLRWQz9G.png
paper_authors.fullname: Alex Zhang
paper_authors.type: user
paper_authors.user: a1zhang
paper_authors._id: 6837123c7b41ef0965ce8f45
paper_authors.name: Thomas L. Griffiths
paper_authors._id: 6837123c7b41ef0965ce8f46
paper_authors.name: Karthik R. Narasimhan
paper_authors._id: 6837123c7b41ef0965ce8f47
paper_authors.name: Ofir Press
paper_title: VideoGameBench: Can Vision-Language Models complete popular video games?
paper_summary: Vision-language models (VLMs) have achieved strong results on coding and math
  benchmarks that are challenging for humans, yet their ability to perform tasks
  that come naturally to humans--such as perception, spatial navigation, and
  memory management--remains understudied. Real video games are crafted to be
  intuitive for humans to learn and master by leveraging innate inductive biases,
  making them an ideal testbed for evaluating such capabilities in VLMs. To this
  end, we introduce VideoGameBench, a benchmark consisting of 10 popular video
  games from the 1990s that VLMs directly interact with in real-time.
  VideoGameBench challenges models to complete entire games with access to only
  raw visual inputs and a high-level description of objectives and controls, a
  significant departure from existing setups that rely on game-specific
  scaffolding and auxiliary information. We keep three of the games secret to
  encourage solutions that generalize to unseen environments. Our experiments
  show that frontier vision-language models struggle to progress beyond the
  beginning of each game. We find inference latency to be a major limitation of
  frontier models in the real-time setting; therefore, we introduce
  VideoGameBench Lite, a setting where the game pauses while waiting for the LM's
  next action. The best performing model, Gemini 2.5 Pro, completes only 0.48% of
  VideoGameBench and 1.6% of VideoGameBench Lite. We hope that the formalization
  of the human skills mentioned above into this benchmark motivates progress in
  these research directions.
paper_ai_summary: VideoGameBench evaluates vision-language models' abilities in real-time video game interaction using only visual inputs and high-level objectives, highlighting challenges in human-like skills.
paper_ai_keywords: vision-language models VideoGameBench real-time interaction visual inputs high-level objectives inference latency VideoGameBench Lite Gemini 2.5 Pro