title: SPRING: GPT-4 Out-performs RL Algorithms by Studying Papers and
    Reasoning
summary: Open-world survival games pose significant challenges for AI algorithms due
  to their multi-tasking, deep exploration, and goal prioritization requirements.
  Despite reinforcement learning (RL) being popular for solving games, its high
  sample complexity limits its effectiveness in complex open-world games like
  Crafter or Minecraft. We propose a novel approach, SPRING, to read the game's
  original academic paper and use the knowledge learned to reason and play the
  game through a large language model (LLM). Prompted with the LaTeX source as
  game context and a description of the agent's current observation, our SPRING
  framework employs a directed acyclic graph (DAG) with game-related questions as
  nodes and dependencies as edges. We identify the optimal action to take in the
  environment by traversing the DAG and calculating LLM responses for each node
  in topological order, with the LLM's answer to final node directly translating
  to environment actions. In our experiments, we study the quality of in-context
  "reasoning" induced by different forms of prompts under the setting of the
  Crafter open-world environment. Our experiments suggest that LLMs, when
  prompted with consistent chain-of-thought, have great potential in completing
  sophisticated high-level trajectories. Quantitatively, SPRING with GPT-4
  outperforms all state-of-the-art RL baselines, trained for 1M steps, without
  any training. Finally, we show the potential of games as a test bed for LLMs.
paper_authors._id: 64700b4128638f11a8400346
paper_authors.name: Yue Wu
paper_authors._id: 64700b4128638f11a8400347
paper_authors.name: So Yeon Min
paper_authors._id: 64700b4128638f11a8400348
paper_authors.name: Shrimai Prabhumoye
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-26T18:35:23.287Z
paper_authors._id: 6470efe7fa9fd77212e17a97
paper_authors.avatarUrl: /avatars/a116b279b927627eeee9c23021b6e5ee.svg
paper_authors.fullname: Shrimai Prabhumoye
paper_authors.type: user
paper_authors.user: shrimai
paper_authors._id: 64700b4128638f11a8400349
paper_authors.name: Yonatan Bisk
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T14:04:29.075Z
paper_authors._id: 63180f3c879073cf83f135b0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63180f3c879073cf83f135b0/0ycXVKOywlU1hcvQ4MDws.jpeg
paper_authors.fullname: Yonatan Bisk
paper_authors.type: user
paper_authors.user: ybisk
paper_authors._id: 64700b4128638f11a840034a
paper_authors.name: Ruslan Salakhutdinov
paper_authors._id: 64700b4128638f11a840034b
paper_authors.name: Amos Azaria
paper_authors._id: 64700b4128638f11a840034c
paper_authors.name: Tom Mitchell
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T14:05:03.692Z
paper_authors._id: 6439ac853ab54fdbab81c5c4
paper_authors.avatarUrl: /avatars/5592fb0420e393a821ae60e6280e1553.svg
paper_authors.fullname: Tom Mitchell
paper_authors.type: user
paper_authors.user: tommmitchell
paper_authors._id: 64700b4128638f11a840034d
paper_authors.name: Yuanzhi Li
paper_title: SPRING: GPT-4 Out-performs RL Algorithms by Studying Papers and
    Reasoning
paper_summary: Open-world survival games pose significant challenges for AI algorithms due
  to their multi-tasking, deep exploration, and goal prioritization requirements.
  Despite reinforcement learning (RL) being popular for solving games, its high
  sample complexity limits its effectiveness in complex open-world games like
  Crafter or Minecraft. We propose a novel approach, SPRING, to read the game's
  original academic paper and use the knowledge learned to reason and play the
  game through a large language model (LLM). Prompted with the LaTeX source as
  game context and a description of the agent's current observation, our SPRING
  framework employs a directed acyclic graph (DAG) with game-related questions as
  nodes and dependencies as edges. We identify the optimal action to take in the
  environment by traversing the DAG and calculating LLM responses for each node
  in topological order, with the LLM's answer to final node directly translating
  to environment actions. In our experiments, we study the quality of in-context
  "reasoning" induced by different forms of prompts under the setting of the
  Crafter open-world environment. Our experiments suggest that LLMs, when
  prompted with consistent chain-of-thought, have great potential in completing
  sophisticated high-level trajectories. Quantitatively, SPRING with GPT-4
  outperforms all state-of-the-art RL baselines, trained for 1M steps, without
  any training. Finally, we show the potential of games as a test bed for LLMs.
paper_ai_summary: SPRING framework uses LLMs and DAGs to reason and play open-world games by processing game prompts, outperforming RL baselines without training.
paper_ai_keywords: reinforcement learning deep exploration goal prioritization large language model directed acyclic graph chain-of-thought in-context reasoning