title: lmgame-Bench: How Good are LLMs at Playing Games?
summary: Playing video games requires perception, memory, and planning, exactly the
  faculties modern large language model (LLM) agents are expected to master. We
  study the major challenges in using popular video games to evaluate modern LLMs
  and find that directly dropping LLMs into games cannot make an effective
  evaluation, for three reasons -- brittle vision perception, prompt sensitivity,
  and potential data contamination. We introduce lmgame-Bench to turn games into
  reliable evaluations. lmgame-Bench features a suite of platformer, puzzle, and
  narrative games delivered through a unified Gym-style API and paired with
  lightweight perception and memory scaffolds, and is designed to stabilize
  prompt variance and remove contamination. Across 13 leading models, we show
  lmgame-Bench is challenging while still separating models well. Correlation
  analysis shows that every game probes a unique blend of capabilities often
  tested in isolation elsewhere. More interestingly, performing reinforcement
  learning on a single game from lmgame-Bench transfers both to unseen games and
  to external planning tasks. Our evaluation code is available at
  https://github.com/lmgame-org/GamingAgent/lmgame-bench.
paper_authors._id: 682e980fb16c79c271babcbd
paper_authors.name: Lanxiang Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-22T07:15:51.281Z
paper_authors._id: 6301d6455e305a35cb0846a7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6301d6455e305a35cb0846a7/aT2AtzRMSY_T3y02MIUap.jpeg
paper_authors.fullname: Lanxiang Hu
paper_authors.type: user
paper_authors.user: Snyhlxde
paper_authors._id: 682e980fb16c79c271babcbe
paper_authors.name: Mingjia Huo
paper_authors._id: 682e980fb16c79c271babcbf
paper_authors.name: Yuxuan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-22T07:15:48.879Z
paper_authors._id: 653200c5d0f5a9e537e76695
paper_authors.avatarUrl: /avatars/d1beb984d91908c03e0daeb77589a475.svg
paper_authors.fullname: Yuxuan Zhang
paper_authors.type: user
paper_authors.user: Yuxuan13
paper_authors._id: 682e980fb16c79c271babcc0
paper_authors.name: Haoyang Yu
paper_authors._id: 682e980fb16c79c271babcc1
paper_authors.name: Eric P. Xing
paper_authors._id: 682e980fb16c79c271babcc2
paper_authors.name: Ion Stoica
paper_authors._id: 682e980fb16c79c271babcc3
paper_authors.name: Tajana Rosing
paper_authors._id: 682e980fb16c79c271babcc4
paper_authors.name: Haojian Jin
paper_authors._id: 682e980fb16c79c271babcc5
paper_authors.name: Hao Zhang
paper_title: lmgame-Bench: How Good are LLMs at Playing Games?
paper_summary: Playing video games requires perception, memory, and planning, exactly the
  faculties modern large language model (LLM) agents are expected to master. We
  study the major challenges in using popular video games to evaluate modern LLMs
  and find that directly dropping LLMs into games cannot make an effective
  evaluation, for three reasons -- brittle vision perception, prompt sensitivity,
  and potential data contamination. We introduce lmgame-Bench to turn games into
  reliable evaluations. lmgame-Bench features a suite of platformer, puzzle, and
  narrative games delivered through a unified Gym-style API and paired with
  lightweight perception and memory scaffolds, and is designed to stabilize
  prompt variance and remove contamination. Across 13 leading models, we show
  lmgame-Bench is challenging while still separating models well. Correlation
  analysis shows that every game probes a unique blend of capabilities often
  tested in isolation elsewhere. More interestingly, performing reinforcement
  learning on a single game from lmgame-Bench transfers both to unseen games and
  to external planning tasks. Our evaluation code is available at
  https://github.com/lmgame-org/GamingAgent/lmgame-bench.
paper_ai_summary: lmgame-Bench evaluates large language models using games with diverse challenges, demonstrating unique capability blends and transfer learning potential.
paper_ai_keywords: LLMs large language model agents brittle vision perception prompt sensitivity data contamination Gym-style API perception scaffolds memory scaffolds reinforcement learning transfer learning