Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on
  Diverse Video Games
Large Language Model (LLM) agents are reshaping the game industry,
particularly with more intelligent and human-preferable game characters.
However, existing game benchmarks fall short of practical needs: they lack
evaluations of diverse LLM capabilities across various game genres, studies of
agentic modules crucial for complex gameplay, and fine-tuning datasets for
aligning pre-trained LLMs into gaming agents. To fill these gaps, we present
\benchname{}, a foundational benchmark designed to train and evaluate
LLM agents across diverse real-world video games. Unlike existing benchmarks,
Orak includes 12 popular video games spanning all major genres, enabling
comprehensive studies of LLM capabilities and agentic modules essential for
intricate game scenarios. To support consistent evaluation of LLMs, we
introduce a plug-and-play interface based on Model Context Protocol (MCP) that
enables LLMs to seamlessly connect with games and manipulate agentic modules.
Additionally, we propose a fine-tuning dataset, consisting of LLM gameplay
trajectories across diverse game genres. Orak offers a comprehensive evaluation
framework, encompassing general game score leaderboards, LLM battle arenas, and
in-depth analyses of visual input state, agentic strategies, and fine-tuning
effects, establishing a foundation towards building generic gaming agents. Code
is available at https://github.com/krafton-ai/Orak.
684194728986af36f6a17bd6
Dongmin Park
684194728986af36f6a17bd7
Minkyu Kim
684194728986af36f6a17bd8
Beongjun Choi
684194728986af36f6a17bd9
Junhyuck Kim
684194728986af36f6a17bda
Keon Lee
684194728986af36f6a17bdb
Jonghyun Lee
684194728986af36f6a17bdc
Inkyu Park
684194728986af36f6a17bdd
Byeong-Uk Lee
684194728986af36f6a17bde
Jaeyoung Hwang
684194728986af36f6a17bdf
Jaewoo Ahn
claimed_verified
2025-06-05T15:07:52.251Z
64bb081c01f1983a863654dc
/avatars/038894bc72b92ec3f4ecb096cc60b60a.svg
Jaewoo Ahn
user
ahnpersie
684194728986af36f6a17be0
Ameya S. Mahabaleshwarkar
684194728986af36f6a17be1
Bilal Kartal
684194728986af36f6a17be2
Pritam Biswas
684194728986af36f6a17be3
Yoshi Suhara
684194728986af36f6a17be4
Kangwook Lee
684194728986af36f6a17be5
Jaewoong Cho
Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on
  Diverse Video Games
Large Language Model (LLM) agents are reshaping the game industry,
particularly with more intelligent and human-preferable game characters.
However, existing game benchmarks fall short of practical needs: they lack
evaluations of diverse LLM capabilities across various game genres, studies of
agentic modules crucial for complex gameplay, and fine-tuning datasets for
aligning pre-trained LLMs into gaming agents. To fill these gaps, we present
\benchname{}, a foundational benchmark designed to train and evaluate
LLM agents across diverse real-world video games. Unlike existing benchmarks,
Orak includes 12 popular video games spanning all major genres, enabling
comprehensive studies of LLM capabilities and agentic modules essential for
intricate game scenarios. To support consistent evaluation of LLMs, we
introduce a plug-and-play interface based on Model Context Protocol (MCP) that
enables LLMs to seamlessly connect with games and manipulate agentic modules.
Additionally, we propose a fine-tuning dataset, consisting of LLM gameplay
trajectories across diverse game genres. Orak offers a comprehensive evaluation
framework, encompassing general game score leaderboards, LLM battle arenas, and
in-depth analyses of visual input state, agentic strategies, and fine-tuning
effects, establishing a foundation towards building generic gaming agents. Code
is available at https://github.com/krafton-ai/Orak.
Orak is a benchmark for training and evaluating LLM agents across diverse video games, featuring a plug-and-play interface and fine-tuning datasets to enhance agentic modules and gameplay.
Large Language Model (LLM) game characters game benchmarks agentic modules fine-tuning datasets Orak Model Context Protocol (MCP) gameplay trajectories game score leaderboards LLM battle arenas visual input state agentic strategies