lmgame-Bench: How Good are LLMs at Playing Games?
Playing video games requires perception, memory, and planning, exactly the
faculties modern large language model (LLM) agents are expected to master. We
study the major challenges in using popular video games to evaluate modern LLMs
and find that directly dropping LLMs into games cannot make an effective
evaluation, for three reasons -- brittle vision perception, prompt sensitivity,
and potential data contamination. We introduce lmgame-Bench to turn games into
reliable evaluations. lmgame-Bench features a suite of platformer, puzzle, and
narrative games delivered through a unified Gym-style API and paired with
lightweight perception and memory scaffolds, and is designed to stabilize
prompt variance and remove contamination. Across 13 leading models, we show
lmgame-Bench is challenging while still separating models well. Correlation
analysis shows that every game probes a unique blend of capabilities often
tested in isolation elsewhere. More interestingly, performing reinforcement
learning on a single game from lmgame-Bench transfers both to unseen games and
to external planning tasks. Our evaluation code is available at
https://github.com/lmgame-org/GamingAgent/lmgame-bench.
682e980fb16c79c271babcbd
Lanxiang Hu
claimed_verified
2025-05-22T07:15:51.281Z
6301d6455e305a35cb0846a7
https://cdn-avatars.huggingface.co/v1/production/uploads/6301d6455e305a35cb0846a7/aT2AtzRMSY_T3y02MIUap.jpeg
Lanxiang Hu
user
Snyhlxde
682e980fb16c79c271babcbe
Mingjia Huo
682e980fb16c79c271babcbf
Yuxuan Zhang
claimed_verified
2025-05-22T07:15:48.879Z
653200c5d0f5a9e537e76695
/avatars/d1beb984d91908c03e0daeb77589a475.svg
Yuxuan Zhang
user
Yuxuan13
682e980fb16c79c271babcc0
Haoyang Yu
682e980fb16c79c271babcc1
Eric P. Xing
682e980fb16c79c271babcc2
Ion Stoica
682e980fb16c79c271babcc3
Tajana Rosing
682e980fb16c79c271babcc4
Haojian Jin
682e980fb16c79c271babcc5
Hao Zhang
lmgame-Bench: How Good are LLMs at Playing Games?
Playing video games requires perception, memory, and planning, exactly the
faculties modern large language model (LLM) agents are expected to master. We
study the major challenges in using popular video games to evaluate modern LLMs
and find that directly dropping LLMs into games cannot make an effective
evaluation, for three reasons -- brittle vision perception, prompt sensitivity,
and potential data contamination. We introduce lmgame-Bench to turn games into
reliable evaluations. lmgame-Bench features a suite of platformer, puzzle, and
narrative games delivered through a unified Gym-style API and paired with
lightweight perception and memory scaffolds, and is designed to stabilize
prompt variance and remove contamination. Across 13 leading models, we show
lmgame-Bench is challenging while still separating models well. Correlation
analysis shows that every game probes a unique blend of capabilities often
tested in isolation elsewhere. More interestingly, performing reinforcement
learning on a single game from lmgame-Bench transfers both to unseen games and
to external planning tasks. Our evaluation code is available at
https://github.com/lmgame-org/GamingAgent/lmgame-bench.
lmgame-Bench evaluates large language models using games with diverse challenges, demonstrating unique capability blends and transfer learning potential.
LLMs large language model agents brittle vision perception prompt sensitivity data contamination Gym-style API perception scaffolds memory scaffolds reinforcement learning transfer learning