title: SmartPlay : A Benchmark for LLMs as Intelligent Agents
summary: Recent large language models (LLMs) have demonstrated great potential toward
  intelligent agents and next-gen automation, but there currently lacks a
  systematic benchmark for evaluating LLMs' abilities as agents. We introduce
  SmartPlay: both a challenging benchmark and a methodology for evaluating LLMs
  as agents. SmartPlay consists of 6 different games, including
  Rock-Paper-Scissors, Tower of Hanoi, Minecraft. Each game features a unique
  setting, providing up to 20 evaluation settings and infinite environment
  variations. Each game in SmartPlay uniquely challenges a subset of 9 important
  capabilities of an intelligent LLM agent, including reasoning with object
  dependencies, planning ahead, spatial reasoning, learning from history, and
  understanding randomness. The distinction between the set of capabilities each
  game test allows us to analyze each capability separately. SmartPlay serves not
  only as a rigorous testing ground for evaluating the overall performance of LLM
  agents but also as a road-map for identifying gaps in current methodologies. We
  release our benchmark at github.com/LLMsmartplay/SmartPlay
paper_authors._id: 651cd98b9a47f703e6b2bee7
paper_authors.name: Yue Wu
paper_authors._id: 651cd98b9a47f703e6b2bee8
paper_authors.name: Xuan Tang
paper_authors._id: 651cd98b9a47f703e6b2bee9
paper_authors.name: Tom M. Mitchell
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-07T14:04:37.122Z
paper_authors._id: 6439ac853ab54fdbab81c5c4
paper_authors.avatarUrl: /avatars/5592fb0420e393a821ae60e6280e1553.svg
paper_authors.fullname: Tom Mitchell
paper_authors.type: user
paper_authors.user: tommmitchell
paper_authors._id: 651cd98b9a47f703e6b2beea
paper_authors.name: Yuanzhi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-07T14:05:47.561Z
paper_authors._id: 641d5a313f04f9bf2da17e8c
paper_authors.avatarUrl: /avatars/4035f73c69ddb045573ad96db4f13f04.svg
paper_authors.fullname: Yuanzhi Li
paper_authors.type: user
paper_authors.user: Uushizhu1234
paper_title: SmartPlay : A Benchmark for LLMs as Intelligent Agents
paper_summary: Recent large language models (LLMs) have demonstrated great potential toward
  intelligent agents and next-gen automation, but there currently lacks a
  systematic benchmark for evaluating LLMs' abilities as agents. We introduce
  SmartPlay: both a challenging benchmark and a methodology for evaluating LLMs
  as agents. SmartPlay consists of 6 different games, including
  Rock-Paper-Scissors, Tower of Hanoi, Minecraft. Each game features a unique
  setting, providing up to 20 evaluation settings and infinite environment
  variations. Each game in SmartPlay uniquely challenges a subset of 9 important
  capabilities of an intelligent LLM agent, including reasoning with object
  dependencies, planning ahead, spatial reasoning, learning from history, and
  understanding randomness. The distinction between the set of capabilities each
  game test allows us to analyze each capability separately. SmartPlay serves not
  only as a rigorous testing ground for evaluating the overall performance of LLM
  agents but also as a road-map for identifying gaps in current methodologies. We
  release our benchmark at github.com/LLMsmartplay/SmartPlay
paper_ai_keywords: large language models (LLMs) intelligent agents next-gen automation benchmark evaluation SmartPlay Rock-Paper-Scissors Tower of Hanoi Minecraft evaluation settings environment variations reasoning with object dependencies planning ahead spatial reasoning learning from history understanding randomness