title: Ghost in the Minecraft: Generally Capable Agents for Open-World
    Enviroments via Large Language Models with Text-based Knowledge and Memory
summary: The captivating realm of Minecraft has attracted substantial research
  interest in recent years, serving as a rich platform for developing intelligent
  agents capable of functioning in open-world environments. However, the current
  research landscape predominantly focuses on specific objectives, such as the
  popular "ObtainDiamond" task, and has not yet shown effective generalization to
  a broader spectrum of tasks. Furthermore, the current leading success rate for
  the "ObtainDiamond" task stands at around 20%, highlighting the limitations of
  Reinforcement Learning (RL) based controllers used in existing methods. To
  tackle these challenges, we introduce Ghost in the Minecraft (GITM), a novel
  framework integrates Large Language Models (LLMs) with text-based knowledge and
  memory, aiming to create Generally Capable Agents (GCAs) in Minecraft. These
  agents, equipped with the logic and common sense capabilities of LLMs, can
  skillfully navigate complex, sparse-reward environments with text-based
  interactions. We develop a set of structured actions and leverage LLMs to
  generate action plans for the agents to execute. The resulting LLM-based agent
  markedly surpasses previous methods, achieving a remarkable improvement of
  +47.5% in success rate on the "ObtainDiamond" task, demonstrating superior
  robustness compared to traditional RL-based controllers. Notably, our agent is
  the first to procure all items in the Minecraft Overworld technology tree,
  demonstrating its extensive capabilities. GITM does not need any GPU for
  training, but a single CPU node with 32 CPU cores is enough. This research
  shows the potential of LLMs in developing capable agents for handling
  long-horizon, complex tasks and adapting to uncertainties in open-world
  environments. See the project website at https://github.com/OpenGVLab/GITM.
paper_authors._id: 64758bcca855203d8fefcde4
paper_authors.name: Xizhou Zhu
paper_authors._id: 64758bcca855203d8fefcde5
paper_authors.name: Yuntao Chen
paper_authors._id: 64758bcca855203d8fefcde6
paper_authors.name: Hao Tian
paper_authors._id: 64758bcca855203d8fefcde7
paper_authors.name: Chenxin Tao
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-05-30T06:32:52.799Z
paper_authors._id: 632dab84fdb35759ea6646a0
paper_authors.avatarUrl: /avatars/857b0b4d115aa5ab2f143e60b0e4edc6.svg
paper_authors.fullname: Yuntao Chen
paper_authors.type: user
paper_authors.user: YuntaoChen
paper_authors._id: 64758bcca855203d8fefcde8
paper_authors.name: Weijie Su
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-30T07:46:41.610Z
paper_authors._id: 63e4562f9db5da2dc1f3b520
paper_authors.avatarUrl: /avatars/f4eecf1396b05e1c72436e7026d85cef.svg
paper_authors.fullname: Weijie Su
paper_authors.type: user
paper_authors.user: jackroos
paper_authors._id: 64758bcca855203d8fefcde9
paper_authors.name: Chenyu Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-30T07:01:02.975Z
paper_authors._id: 637f347a52229c639211bee8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637f347a52229c639211bee8/I9_PET-_6SJQJ6hXrACV4.jpeg
paper_authors.fullname: Chenyu Yang
paper_authors.type: user
paper_authors.user: cyyang822
paper_authors._id: 64758bcca855203d8fefcdea
paper_authors.name: Gao Huang
paper_authors._id: 64758bcca855203d8fefcdeb
paper_authors.name: Bin Li
paper_authors._id: 64758bcca855203d8fefcdec
paper_authors.name: Lewei Lu
paper_authors._id: 64758bcca855203d8fefcded
paper_authors.name: Xiaogang Wang
paper_authors._id: 64758bcca855203d8fefcdee
paper_authors.name: Yu Qiao
paper_authors._id: 64758bcca855203d8fefcdef
paper_authors.name: Zhaoxiang Zhang
paper_authors._id: 64758bcca855203d8fefcdf0
paper_authors.name: Jifeng Dai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-30T07:46:44.632Z
paper_authors._id: 64686f7172d9180d4ac8b4e4
paper_authors.avatarUrl: /avatars/db67dd6c4b2b41054ddcce5a18ade6f8.svg
paper_authors.fullname: Jifeng Dai
paper_authors.type: user
paper_authors.user: daijifeng
paper_title: Ghost in the Minecraft: Generally Capable Agents for Open-World
    Enviroments via Large Language Models with Text-based Knowledge and Memory
paper_summary: The captivating realm of Minecraft has attracted substantial research
  interest in recent years, serving as a rich platform for developing intelligent
  agents capable of functioning in open-world environments. However, the current
  research landscape predominantly focuses on specific objectives, such as the
  popular "ObtainDiamond" task, and has not yet shown effective generalization to
  a broader spectrum of tasks. Furthermore, the current leading success rate for
  the "ObtainDiamond" task stands at around 20%, highlighting the limitations of
  Reinforcement Learning (RL) based controllers used in existing methods. To
  tackle these challenges, we introduce Ghost in the Minecraft (GITM), a novel
  framework integrates Large Language Models (LLMs) with text-based knowledge and
  memory, aiming to create Generally Capable Agents (GCAs) in Minecraft. These
  agents, equipped with the logic and common sense capabilities of LLMs, can
  skillfully navigate complex, sparse-reward environments with text-based
  interactions. We develop a set of structured actions and leverage LLMs to
  generate action plans for the agents to execute. The resulting LLM-based agent
  markedly surpasses previous methods, achieving a remarkable improvement of
  +47.5% in success rate on the "ObtainDiamond" task, demonstrating superior
  robustness compared to traditional RL-based controllers. Notably, our agent is
  the first to procure all items in the Minecraft Overworld technology tree,
  demonstrating its extensive capabilities. GITM does not need any GPU for
  training, but a single CPU node with 32 CPU cores is enough. This research
  shows the potential of LLMs in developing capable agents for handling
  long-horizon, complex tasks and adapting to uncertainties in open-world
  environments. See the project website at https://github.com/OpenGVLab/GITM.
paper_ai_summary: A framework integrating Large Language Models with text-based knowledge achieves superior performance in Minecraft by surpassing traditional RL methods and completing complex tasks.
paper_ai_keywords: Large Language Models text-based knowledge memory Generally Capable Agents action plans long-horizon tasks