title: BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games
summary: Large Language Models (LLMs) and Vision Language Models (VLMs) possess
  extensive knowledge and exhibit promising reasoning abilities; however, they
  still struggle to perform well in complex, dynamic environments. Real-world
  tasks require handling intricate interactions, advanced spatial reasoning,
  long-term planning, and continuous exploration of new strategies-areas in which
  we lack effective methodologies for comprehensively evaluating these
  capabilities. To address this gap, we introduce BALROG, a novel benchmark
  designed to assess the agentic capabilities of LLMs and VLMs through a diverse
  set of challenging games. Our benchmark incorporates a range of existing
  reinforcement learning environments with varying levels of difficulty,
  including tasks that are solvable by non-expert humans in seconds to extremely
  challenging ones that may take years to master (e.g., the NetHack Learning
  Environment). We devise fine-grained metrics to measure performance and conduct
  an extensive evaluation of several popular open-source and closed-source LLMs
  and VLMs. Our findings indicate that while current models achieve partial
  success in the easier games, they struggle significantly with more challenging
  tasks. Notably, we observe severe deficiencies in vision-based decision-making,
  as models perform worse when visual representations of the environments are
  provided. We release BALROG as an open and user-friendly benchmark to
  facilitate future research and development in the agentic community.
paper_authors._id: 673fa529128c57b2f2996fdc
paper_authors.name: Davide Paglieri
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-21T21:32:39.052Z
paper_authors._id: 6564d9a83f025ef9f11b28bb
paper_authors.avatarUrl: /avatars/ab19f5f339e80c3b0a4b8d70abaced94.svg
paper_authors.fullname: Davide Paglieri
paper_authors.type: user
paper_authors.user: pagli98
paper_authors._id: 673fa529128c57b2f2996fdd
paper_authors.name: Bartłomiej Cupiał
paper_authors._id: 673fa529128c57b2f2996fde
paper_authors.name: Samuel Coward
paper_authors._id: 673fa529128c57b2f2996fdf
paper_authors.name: Ulyana Piterbarg
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:20:00.560Z
paper_authors._id: 6337054e0267ebcf02637018
paper_authors.avatarUrl: /avatars/89ad12d6643f18706050a92efccd92e6.svg
paper_authors.fullname: Ulyana Piterbarg
paper_authors.type: user
paper_authors.user: upiter
paper_authors._id: 673fa529128c57b2f2996fe0
paper_authors.name: Maciej Wolczyk
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:19:55.403Z
paper_authors._id: 62a4ffc96ee324b134d136fc
paper_authors.avatarUrl: /avatars/c674bb48a85b0347c636bc548bdc3ad4.svg
paper_authors.fullname: Maciej Wolczyk
paper_authors.type: user
paper_authors.user: rahid
paper_authors._id: 673fa529128c57b2f2996fe1
paper_authors.name: Akbir Khan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:19:50.327Z
paper_authors._id: 62a0788fd066f825a0164c68
paper_authors.avatarUrl: /avatars/a565c173c82dea26a5a61370ce42b81f.svg
paper_authors.fullname: Akbir Khan
paper_authors.type: user
paper_authors.user: akbir
paper_authors._id: 673fa529128c57b2f2996fe2
paper_authors.name: Eduardo Pignatelli
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:19:44.723Z
paper_authors._id: 630499ba34c824b1724ca3ef
paper_authors.avatarUrl: /avatars/a4cae175448775d9505f8d47eefe5fe1.svg
paper_authors.fullname: Eduardo Pignatelli
paper_authors.type: user
paper_authors.user: epignatelli
paper_authors._id: 673fa529128c57b2f2996fe3
paper_authors.name: Łukasz Kuciński
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:19:39.537Z
paper_authors._id: 6580724062b6ed53c4c377cd
paper_authors.avatarUrl: /avatars/b5bd5e9e7399a47bf5ec5e3ce9db9307.svg
paper_authors.fullname: Łukasz Kuciński
paper_authors.type: user
paper_authors.user: lukaszkucinski
paper_authors._id: 673fa529128c57b2f2996fe4
paper_authors.name: Lerrel Pinto
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-25T21:19:33.209Z
paper_authors._id: 66fffa0766bc54f4e532e3d2
paper_authors.avatarUrl: /avatars/9db8b2183097bcaddded06d1b800cf77.svg
paper_authors.fullname: Lerrel Pinto
paper_authors.type: user
paper_authors.user: LerrelPinto
paper_authors._id: 673fa529128c57b2f2996fe5
paper_authors.name: Rob Fergus
paper_authors._id: 673fa529128c57b2f2996fe6
paper_authors.name: Jakob Nicolaus Foerster
paper_authors._id: 673fa529128c57b2f2996fe7
paper_authors.name: Jack Parker-Holder
paper_authors._id: 673fa529128c57b2f2996fe8
paper_authors.name: Tim Rocktäschel
paper_title: BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games
paper_summary: Large Language Models (LLMs) and Vision Language Models (VLMs) possess
  extensive knowledge and exhibit promising reasoning abilities; however, they
  still struggle to perform well in complex, dynamic environments. Real-world
  tasks require handling intricate interactions, advanced spatial reasoning,
  long-term planning, and continuous exploration of new strategies-areas in which
  we lack effective methodologies for comprehensively evaluating these
  capabilities. To address this gap, we introduce BALROG, a novel benchmark
  designed to assess the agentic capabilities of LLMs and VLMs through a diverse
  set of challenging games. Our benchmark incorporates a range of existing
  reinforcement learning environments with varying levels of difficulty,
  including tasks that are solvable by non-expert humans in seconds to extremely
  challenging ones that may take years to master (e.g., the NetHack Learning
  Environment). We devise fine-grained metrics to measure performance and conduct
  an extensive evaluation of several popular open-source and closed-source LLMs
  and VLMs. Our findings indicate that while current models achieve partial
  success in the easier games, they struggle significantly with more challenging
  tasks. Notably, we observe severe deficiencies in vision-based decision-making,
  as models perform worse when visual representations of the environments are
  provided. We release BALROG as an open and user-friendly benchmark to
  facilitate future research and development in the agentic community.
paper_ai_keywords: Large Language Models (LLMs) Vision Language Models (VLMs) reasoning abilities complex, dynamic environments intricate interactions advanced spatial reasoning long-term planning continuous exploration methodologies BALROG benchmark agentic capabilities reinforcement learning environments NetHack Learning Environment fine-grained metrics vision-based decision-making