title: Measuring General Intelligence with Generated Games
summary: We present gg-bench, a collection of game environments designed to evaluate
  general reasoning capabilities in language models. Unlike most static
  benchmarks, gg-bench is a data generating process where new evaluation
  instances can be generated at will. In particular, gg-bench is synthetically
  generated by (1) using a large language model (LLM) to generate natural
  language descriptions of novel games, (2) using the LLM to implement each game
  in code as a Gym environment, and (3) training reinforcement learning (RL)
  agents via self-play on the generated games. We evaluate language models by
  their winrate against these RL agents by prompting models with the game
  description, current board state, and a list of valid moves, after which models
  output the moves they wish to take. gg-bench is challenging: state-of-the-art
  LLMs such as GPT-4o and Claude 3.7 Sonnet achieve winrates of 7-9% on gg-bench
  using in-context learning, while reasoning models such as o1, o3-mini and
  DeepSeek-R1 achieve average winrates of 31-36%. We release the generated games,
  data generation process, and evaluation code in order to support future
  modeling work and expansion of our benchmark.
paper_authors._id: 68236b86102b1d3069ebafab
paper_authors.name: Vivek Verma
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-05-13T15:55:50.678Z
paper_authors._id: 64b88247e436bbca16603baf
paper_authors.avatarUrl: /avatars/7bde6b0f75bccc3195fb72cbe5860a7e.svg
paper_authors.fullname: Vivek Verma
paper_authors.type: user
paper_authors.user: vivekverma
paper_authors._id: 68236b86102b1d3069ebafac
paper_authors.name: David Huang
paper_authors._id: 68236b86102b1d3069ebafad
paper_authors.name: William Chen
paper_authors._id: 68236b86102b1d3069ebafae
paper_authors.name: Dan Klein
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-14T10:09:04.358Z
paper_authors._id: 632be88b3690fb57e70e0bf1
paper_authors.avatarUrl: /avatars/74ff8f30b3662db2602495bdf493d397.svg
paper_authors.fullname: Dan Klein
paper_authors.type: user
paper_authors.user: danjklein
paper_authors._id: 68236b86102b1d3069ebafaf
paper_authors.name: Nicholas Tomlin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-14T07:35:26.733Z
paper_authors._id: 6269d074a6a7bba9e46d8d50
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1651101782106-noauth.jpeg
paper_authors.fullname: Nicholas Tomlin
paper_authors.type: user
paper_authors.user: nickatomlin
paper_title: Measuring General Intelligence with Generated Games
paper_summary: We present gg-bench, a collection of game environments designed to evaluate
  general reasoning capabilities in language models. Unlike most static
  benchmarks, gg-bench is a data generating process where new evaluation
  instances can be generated at will. In particular, gg-bench is synthetically
  generated by (1) using a large language model (LLM) to generate natural
  language descriptions of novel games, (2) using the LLM to implement each game
  in code as a Gym environment, and (3) training reinforcement learning (RL)
  agents via self-play on the generated games. We evaluate language models by
  their winrate against these RL agents by prompting models with the game
  description, current board state, and a list of valid moves, after which models
  output the moves they wish to take. gg-bench is challenging: state-of-the-art
  LLMs such as GPT-4o and Claude 3.7 Sonnet achieve winrates of 7-9% on gg-bench
  using in-context learning, while reasoning models such as o1, o3-mini and
  DeepSeek-R1 achieve average winrates of 31-36%. We release the generated games,
  data generation process, and evaluation code in order to support future
  modeling work and expansion of our benchmark.
paper_ai_summary: gg-bench is a dynamic benchmark for evaluating general reasoning in language models by generating new game environments and assessing winrates against reinforcement learning agents.
paper_ai_keywords: game environments general reasoning capabilities language models data generating process natural language descriptions Gym environment reinforcement learning RL agents self-play winrate in-context learning reasoning models GPT-4o Claude 3.7 Sonnet o1 o3-mini DeepSeek-R1