SPRING: GPT-4 Out-performs RL Algorithms by Studying Papers and
  Reasoning
Open-world survival games pose significant challenges for AI algorithms due
to their multi-tasking, deep exploration, and goal prioritization requirements.
Despite reinforcement learning (RL) being popular for solving games, its high
sample complexity limits its effectiveness in complex open-world games like
Crafter or Minecraft. We propose a novel approach, SPRING, to read the game's
original academic paper and use the knowledge learned to reason and play the
game through a large language model (LLM). Prompted with the LaTeX source as
game context and a description of the agent's current observation, our SPRING
framework employs a directed acyclic graph (DAG) with game-related questions as
nodes and dependencies as edges. We identify the optimal action to take in the
environment by traversing the DAG and calculating LLM responses for each node
in topological order, with the LLM's answer to final node directly translating
to environment actions. In our experiments, we study the quality of in-context
"reasoning" induced by different forms of prompts under the setting of the
Crafter open-world environment. Our experiments suggest that LLMs, when
prompted with consistent chain-of-thought, have great potential in completing
sophisticated high-level trajectories. Quantitatively, SPRING with GPT-4
outperforms all state-of-the-art RL baselines, trained for 1M steps, without
any training. Finally, we show the potential of games as a test bed for LLMs.
64700b4128638f11a8400346
Yue Wu
64700b4128638f11a8400347
So Yeon Min
64700b4128638f11a8400348
Shrimai Prabhumoye
claimed_verified
2023-05-26T18:35:23.287Z
6470efe7fa9fd77212e17a97
/avatars/a116b279b927627eeee9c23021b6e5ee.svg
Shrimai Prabhumoye
user
shrimai
64700b4128638f11a8400349
Yonatan Bisk
admin_assigned
2023-05-26T14:04:29.075Z
63180f3c879073cf83f135b0
https://cdn-avatars.huggingface.co/v1/production/uploads/63180f3c879073cf83f135b0/0ycXVKOywlU1hcvQ4MDws.jpeg
Yonatan Bisk
user
ybisk
64700b4128638f11a840034a
Ruslan Salakhutdinov
64700b4128638f11a840034b
Amos Azaria
64700b4128638f11a840034c
Tom Mitchell
admin_assigned
2023-05-26T14:05:03.692Z
6439ac853ab54fdbab81c5c4
/avatars/5592fb0420e393a821ae60e6280e1553.svg
Tom Mitchell
user
tommmitchell
64700b4128638f11a840034d
Yuanzhi Li
SPRING: GPT-4 Out-performs RL Algorithms by Studying Papers and
  Reasoning
Open-world survival games pose significant challenges for AI algorithms due
to their multi-tasking, deep exploration, and goal prioritization requirements.
Despite reinforcement learning (RL) being popular for solving games, its high
sample complexity limits its effectiveness in complex open-world games like
Crafter or Minecraft. We propose a novel approach, SPRING, to read the game's
original academic paper and use the knowledge learned to reason and play the
game through a large language model (LLM). Prompted with the LaTeX source as
game context and a description of the agent's current observation, our SPRING
framework employs a directed acyclic graph (DAG) with game-related questions as
nodes and dependencies as edges. We identify the optimal action to take in the
environment by traversing the DAG and calculating LLM responses for each node
in topological order, with the LLM's answer to final node directly translating
to environment actions. In our experiments, we study the quality of in-context
"reasoning" induced by different forms of prompts under the setting of the
Crafter open-world environment. Our experiments suggest that LLMs, when
prompted with consistent chain-of-thought, have great potential in completing
sophisticated high-level trajectories. Quantitatively, SPRING with GPT-4
outperforms all state-of-the-art RL baselines, trained for 1M steps, without
any training. Finally, we show the potential of games as a test bed for LLMs.
SPRING framework uses LLMs and DAGs to reason and play open-world games by processing game prompts, outperforming RL baselines without training.
reinforcement learning deep exploration goal prioritization large language model directed acyclic graph chain-of-thought in-context reasoning