title: Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on
    Diverse Video Games
summary: Large Language Model (LLM) agents are reshaping the game industry,
  particularly with more intelligent and human-preferable game characters.
  However, existing game benchmarks fall short of practical needs: they lack
  evaluations of diverse LLM capabilities across various game genres, studies of
  agentic modules crucial for complex gameplay, and fine-tuning datasets for
  aligning pre-trained LLMs into gaming agents. To fill these gaps, we present
  \benchname{}, a foundational benchmark designed to train and evaluate
  LLM agents across diverse real-world video games. Unlike existing benchmarks,
  Orak includes 12 popular video games spanning all major genres, enabling
  comprehensive studies of LLM capabilities and agentic modules essential for
  intricate game scenarios. To support consistent evaluation of LLMs, we
  introduce a plug-and-play interface based on Model Context Protocol (MCP) that
  enables LLMs to seamlessly connect with games and manipulate agentic modules.
  Additionally, we propose a fine-tuning dataset, consisting of LLM gameplay
  trajectories across diverse game genres. Orak offers a comprehensive evaluation
  framework, encompassing general game score leaderboards, LLM battle arenas, and
  in-depth analyses of visual input state, agentic strategies, and fine-tuning
  effects, establishing a foundation towards building generic gaming agents. Code
  is available at https://github.com/krafton-ai/Orak.
paper_authors._id: 684194728986af36f6a17bd6
paper_authors.name: Dongmin Park
paper_authors._id: 684194728986af36f6a17bd7
paper_authors.name: Minkyu Kim
paper_authors._id: 684194728986af36f6a17bd8
paper_authors.name: Beongjun Choi
paper_authors._id: 684194728986af36f6a17bd9
paper_authors.name: Junhyuck Kim
paper_authors._id: 684194728986af36f6a17bda
paper_authors.name: Keon Lee
paper_authors._id: 684194728986af36f6a17bdb
paper_authors.name: Jonghyun Lee
paper_authors._id: 684194728986af36f6a17bdc
paper_authors.name: Inkyu Park
paper_authors._id: 684194728986af36f6a17bdd
paper_authors.name: Byeong-Uk Lee
paper_authors._id: 684194728986af36f6a17bde
paper_authors.name: Jaeyoung Hwang
paper_authors._id: 684194728986af36f6a17bdf
paper_authors.name: Jaewoo Ahn
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-05T15:07:52.251Z
paper_authors._id: 64bb081c01f1983a863654dc
paper_authors.avatarUrl: /avatars/038894bc72b92ec3f4ecb096cc60b60a.svg
paper_authors.fullname: Jaewoo Ahn
paper_authors.type: user
paper_authors.user: ahnpersie
paper_authors._id: 684194728986af36f6a17be0
paper_authors.name: Ameya S. Mahabaleshwarkar
paper_authors._id: 684194728986af36f6a17be1
paper_authors.name: Bilal Kartal
paper_authors._id: 684194728986af36f6a17be2
paper_authors.name: Pritam Biswas
paper_authors._id: 684194728986af36f6a17be3
paper_authors.name: Yoshi Suhara
paper_authors._id: 684194728986af36f6a17be4
paper_authors.name: Kangwook Lee
paper_authors._id: 684194728986af36f6a17be5
paper_authors.name: Jaewoong Cho
paper_title: Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on
    Diverse Video Games
paper_summary: Large Language Model (LLM) agents are reshaping the game industry,
  particularly with more intelligent and human-preferable game characters.
  However, existing game benchmarks fall short of practical needs: they lack
  evaluations of diverse LLM capabilities across various game genres, studies of
  agentic modules crucial for complex gameplay, and fine-tuning datasets for
  aligning pre-trained LLMs into gaming agents. To fill these gaps, we present
  \benchname{}, a foundational benchmark designed to train and evaluate
  LLM agents across diverse real-world video games. Unlike existing benchmarks,
  Orak includes 12 popular video games spanning all major genres, enabling
  comprehensive studies of LLM capabilities and agentic modules essential for
  intricate game scenarios. To support consistent evaluation of LLMs, we
  introduce a plug-and-play interface based on Model Context Protocol (MCP) that
  enables LLMs to seamlessly connect with games and manipulate agentic modules.
  Additionally, we propose a fine-tuning dataset, consisting of LLM gameplay
  trajectories across diverse game genres. Orak offers a comprehensive evaluation
  framework, encompassing general game score leaderboards, LLM battle arenas, and
  in-depth analyses of visual input state, agentic strategies, and fine-tuning
  effects, establishing a foundation towards building generic gaming agents. Code
  is available at https://github.com/krafton-ai/Orak.
paper_ai_summary: Orak is a benchmark for training and evaluating LLM agents across diverse video games, featuring a plug-and-play interface and fine-tuning datasets to enhance agentic modules and gameplay.
paper_ai_keywords: Large Language Model (LLM) game characters game benchmarks agentic modules fine-tuning datasets Orak Model Context Protocol (MCP) gameplay trajectories game score leaderboards LLM battle arenas visual input state agentic strategies