title: JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal
    Language Models
summary: Achieving human-like planning and control with multimodal observations in an
  open world is a key milestone for more functional generalist agents. Existing
  approaches can handle certain long-horizon tasks in an open world. However,
  they still struggle when the number of open-world tasks could potentially be
  infinite and lack the capability to progressively enhance task completion as
  game time progresses. We introduce JARVIS-1, an open-world agent that can
  perceive multimodal input (visual observations and human instructions),
  generate sophisticated plans, and perform embodied control, all within the
  popular yet challenging open-world Minecraft universe. Specifically, we develop
  JARVIS-1 on top of pre-trained multimodal language models, which map visual
  observations and textual instructions to plans. The plans will be ultimately
  dispatched to the goal-conditioned controllers. We outfit JARVIS-1 with a
  multimodal memory, which facilitates planning using both pre-trained knowledge
  and its actual game survival experiences. In our experiments, JARVIS-1 exhibits
  nearly perfect performances across over 200 varying tasks from the Minecraft
  Universe Benchmark, ranging from entry to intermediate levels. JARVIS-1 has
  achieved a completion rate of 12.5% in the long-horizon diamond pickaxe task.
  This represents a significant increase up to 5 times compared to previous
  records. Furthermore, we show that JARVIS-1 is able to self-improve
  following a life-long learning paradigm thanks to multimodal memory, sparking a
  more general intelligence and improved autonomy. The project page is available
  at https://craftjarvis-jarvis1.github.io.
paper_authors._id: 65519bce32f278f503b336ff
paper_authors.name: Zihao Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-05T13:16:14.374Z
paper_authors._id: 642e8c99c1b0f8e4e76bcaab
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642e8c99c1b0f8e4e76bcaab/BOs9r0P9KyT9pEba9v0H4.png
paper_authors.fullname: Zihao Wang
paper_authors.type: user
paper_authors.user: zhwang4ai
paper_authors._id: 65519bce32f278f503b33700
paper_authors.name: Shaofei Cai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-03T07:42:39.728Z
paper_authors._id: 6578459d62d3ac1817ed79fe
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6578459d62d3ac1817ed79fe/AXDJuwLUoEOb4Fj3U0Xxo.jpeg
paper_authors.fullname: Shaofei Cai
paper_authors.type: user
paper_authors.user: phython96
paper_authors._id: 65519bce32f278f503b33701
paper_authors.name: Anji Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T15:16:28.982Z
paper_authors._id: 66757955b3882fd587d5f363
paper_authors.avatarUrl: /avatars/41fdfa19057ad3ea5ece29b94f163218.svg
paper_authors.fullname: Anji Liu
paper_authors.type: user
paper_authors.user: anjiliu
paper_authors._id: 65519bce32f278f503b33702
paper_authors.name: Yonggang Jin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-13T09:10:11.688Z
paper_authors._id: 648d9106941869a01cfb526e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648d9106941869a01cfb526e/wt1PWbfEuA5cU_blssLWZ.jpeg
paper_authors.fullname: Yonggang Jin
paper_authors.type: user
paper_authors.user: ygjin11
paper_authors._id: 65519bce32f278f503b33703
paper_authors.name: Jinbing Hou
paper_authors._id: 65519bce32f278f503b33704
paper_authors.name: Bowei Zhang
paper_authors._id: 65519bce32f278f503b33705
paper_authors.name: Haowei Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-08T07:03:26.384Z
paper_authors._id: 63453f02a05b51f7ded3c579
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/u7p-wTj8zyeWpSFRI8Go0.png
paper_authors.fullname: Andy Lin
paper_authors.type: user
paper_authors.user: pkuHaowei
paper_authors._id: 65519bce32f278f503b33706
paper_authors.name: Zhaofeng He
paper_authors._id: 65519bce32f278f503b33707
paper_authors.name: Zilong Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-13T09:11:24.628Z
paper_authors._id: 63a95a6a7930fa8c7dd63d4e
paper_authors.avatarUrl: /avatars/d9d0420f7ddfe2f3a7e029fb05f1c89f.svg
paper_authors.fullname: Zilong Zheng
paper_authors.type: user
paper_authors.user: zlzheng
paper_authors._id: 65519bce32f278f503b33708
paper_authors.name: Yaodong Yang
paper_authors._id: 65519bce32f278f503b33709
paper_authors.name: Xiaojian Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-13T09:12:01.241Z
paper_authors._id: 60dd0e36a15ddd7d2006d2e9
paper_authors.avatarUrl: /avatars/8bd98177a79efbf295be8f6457683297.svg
paper_authors.fullname: Xiaojian Ma
paper_authors.type: user
paper_authors.user: jeasinema
paper_authors._id: 65519bce32f278f503b3370a
paper_authors.name: Yitao Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-13T09:12:10.164Z
paper_authors._id: 64683a5776bb704aa14588b7
paper_authors.avatarUrl: /avatars/e532756f52c5b95981470ace41a10556.svg
paper_authors.fullname: Yitao Liang
paper_authors.type: user
paper_authors.user: YitaoLiang
paper_title: JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal
    Language Models
paper_summary: Achieving human-like planning and control with multimodal observations in an
  open world is a key milestone for more functional generalist agents. Existing
  approaches can handle certain long-horizon tasks in an open world. However,
  they still struggle when the number of open-world tasks could potentially be
  infinite and lack the capability to progressively enhance task completion as
  game time progresses. We introduce JARVIS-1, an open-world agent that can
  perceive multimodal input (visual observations and human instructions),
  generate sophisticated plans, and perform embodied control, all within the
  popular yet challenging open-world Minecraft universe. Specifically, we develop
  JARVIS-1 on top of pre-trained multimodal language models, which map visual
  observations and textual instructions to plans. The plans will be ultimately
  dispatched to the goal-conditioned controllers. We outfit JARVIS-1 with a
  multimodal memory, which facilitates planning using both pre-trained knowledge
  and its actual game survival experiences. In our experiments, JARVIS-1 exhibits
  nearly perfect performances across over 200 varying tasks from the Minecraft
  Universe Benchmark, ranging from entry to intermediate levels. JARVIS-1 has
  achieved a completion rate of 12.5% in the long-horizon diamond pickaxe task.
  This represents a significant increase up to 5 times compared to previous
  records. Furthermore, we show that JARVIS-1 is able to self-improve
  following a life-long learning paradigm thanks to multimodal memory, sparking a
  more general intelligence and improved autonomy. The project page is available
  at https://craftjarvis-jarvis1.github.io.
paper_ai_summary: JARVIS-1, an open-world agent in Minecraft, uses multimodal perception and a memory system to perform complex tasks and improve over time.
paper_ai_keywords: multimodal observations JARVIS-1 pre-trained multimodal language models multimodal memory goal-conditioned controllers Minecraft Universe Benchmark long-horizon diamond pickaxe task self-improve life-long learning general intelligence autonomy