BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games
Large Language Models (LLMs) and Vision Language Models (VLMs) possess
extensive knowledge and exhibit promising reasoning abilities; however, they
still struggle to perform well in complex, dynamic environments. Real-world
tasks require handling intricate interactions, advanced spatial reasoning,
long-term planning, and continuous exploration of new strategies-areas in which
we lack effective methodologies for comprehensively evaluating these
capabilities. To address this gap, we introduce BALROG, a novel benchmark
designed to assess the agentic capabilities of LLMs and VLMs through a diverse
set of challenging games. Our benchmark incorporates a range of existing
reinforcement learning environments with varying levels of difficulty,
including tasks that are solvable by non-expert humans in seconds to extremely
challenging ones that may take years to master (e.g., the NetHack Learning
Environment). We devise fine-grained metrics to measure performance and conduct
an extensive evaluation of several popular open-source and closed-source LLMs
and VLMs. Our findings indicate that while current models achieve partial
success in the easier games, they struggle significantly with more challenging
tasks. Notably, we observe severe deficiencies in vision-based decision-making,
as models perform worse when visual representations of the environments are
provided. We release BALROG as an open and user-friendly benchmark to
facilitate future research and development in the agentic community.
673fa529128c57b2f2996fdc
Davide Paglieri
claimed_verified
2024-11-21T21:32:39.052Z
6564d9a83f025ef9f11b28bb
/avatars/ab19f5f339e80c3b0a4b8d70abaced94.svg
Davide Paglieri
user
pagli98
673fa529128c57b2f2996fdd
Bartłomiej Cupiał
673fa529128c57b2f2996fde
Samuel Coward
673fa529128c57b2f2996fdf
Ulyana Piterbarg
admin_assigned
2024-11-25T21:20:00.560Z
6337054e0267ebcf02637018
/avatars/89ad12d6643f18706050a92efccd92e6.svg
Ulyana Piterbarg
user
upiter
673fa529128c57b2f2996fe0
Maciej Wolczyk
admin_assigned
2024-11-25T21:19:55.403Z
62a4ffc96ee324b134d136fc
/avatars/c674bb48a85b0347c636bc548bdc3ad4.svg
Maciej Wolczyk
user
rahid
673fa529128c57b2f2996fe1
Akbir Khan
admin_assigned
2024-11-25T21:19:50.327Z
62a0788fd066f825a0164c68
/avatars/a565c173c82dea26a5a61370ce42b81f.svg
Akbir Khan
user
akbir
673fa529128c57b2f2996fe2
Eduardo Pignatelli
admin_assigned
2024-11-25T21:19:44.723Z
630499ba34c824b1724ca3ef
/avatars/a4cae175448775d9505f8d47eefe5fe1.svg
Eduardo Pignatelli
user
epignatelli
673fa529128c57b2f2996fe3
Łukasz Kuciński
admin_assigned
2024-11-25T21:19:39.537Z
6580724062b6ed53c4c377cd
/avatars/b5bd5e9e7399a47bf5ec5e3ce9db9307.svg
Łukasz Kuciński
user
lukaszkucinski
673fa529128c57b2f2996fe4
Lerrel Pinto
admin_assigned
2024-11-25T21:19:33.209Z
66fffa0766bc54f4e532e3d2
/avatars/9db8b2183097bcaddded06d1b800cf77.svg
Lerrel Pinto
user
LerrelPinto
673fa529128c57b2f2996fe5
Rob Fergus
673fa529128c57b2f2996fe6
Jakob Nicolaus Foerster
673fa529128c57b2f2996fe7
Jack Parker-Holder
673fa529128c57b2f2996fe8
Tim Rocktäschel
BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games
Large Language Models (LLMs) and Vision Language Models (VLMs) possess
extensive knowledge and exhibit promising reasoning abilities; however, they
still struggle to perform well in complex, dynamic environments. Real-world
tasks require handling intricate interactions, advanced spatial reasoning,
long-term planning, and continuous exploration of new strategies-areas in which
we lack effective methodologies for comprehensively evaluating these
capabilities. To address this gap, we introduce BALROG, a novel benchmark
designed to assess the agentic capabilities of LLMs and VLMs through a diverse
set of challenging games. Our benchmark incorporates a range of existing
reinforcement learning environments with varying levels of difficulty,
including tasks that are solvable by non-expert humans in seconds to extremely
challenging ones that may take years to master (e.g., the NetHack Learning
Environment). We devise fine-grained metrics to measure performance and conduct
an extensive evaluation of several popular open-source and closed-source LLMs
and VLMs. Our findings indicate that while current models achieve partial
success in the easier games, they struggle significantly with more challenging
tasks. Notably, we observe severe deficiencies in vision-based decision-making,
as models perform worse when visual representations of the environments are
provided. We release BALROG as an open and user-friendly benchmark to
facilitate future research and development in the agentic community.
BALROG, a novel benchmark, evaluates the agentic capabilities of large language models and vision language models across diverse, challenging games with fine-grained metrics, revealing significant performance gaps, especially in vision-based decision-making.
Large Language Models Vision Language Models BALROG benchmark reinforcement learning environments NetHack Learning Environment fine-grained metrics vision-based decision-making