JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal
  Language Models
Achieving human-like planning and control with multimodal observations in an
open world is a key milestone for more functional generalist agents. Existing
approaches can handle certain long-horizon tasks in an open world. However,
they still struggle when the number of open-world tasks could potentially be
infinite and lack the capability to progressively enhance task completion as
game time progresses. We introduce JARVIS-1, an open-world agent that can
perceive multimodal input (visual observations and human instructions),
generate sophisticated plans, and perform embodied control, all within the
popular yet challenging open-world Minecraft universe. Specifically, we develop
JARVIS-1 on top of pre-trained multimodal language models, which map visual
observations and textual instructions to plans. The plans will be ultimately
dispatched to the goal-conditioned controllers. We outfit JARVIS-1 with a
multimodal memory, which facilitates planning using both pre-trained knowledge
and its actual game survival experiences. In our experiments, JARVIS-1 exhibits
nearly perfect performances across over 200 varying tasks from the Minecraft
Universe Benchmark, ranging from entry to intermediate levels. JARVIS-1 has
achieved a completion rate of 12.5% in the long-horizon diamond pickaxe task.
This represents a significant increase up to 5 times compared to previous
records. Furthermore, we show that JARVIS-1 is able to self-improve
following a life-long learning paradigm thanks to multimodal memory, sparking a
more general intelligence and improved autonomy. The project page is available
at https://craftjarvis-jarvis1.github.io.
65519bce32f278f503b336ff
Zihao Wang
claimed_verified
2024-03-05T13:16:14.374Z
642e8c99c1b0f8e4e76bcaab
https://cdn-avatars.huggingface.co/v1/production/uploads/642e8c99c1b0f8e4e76bcaab/BOs9r0P9KyT9pEba9v0H4.png
Zihao Wang
user
zhwang4ai
65519bce32f278f503b33700
Shaofei Cai
claimed_verified
2024-07-03T07:42:39.728Z
6578459d62d3ac1817ed79fe
https://cdn-avatars.huggingface.co/v1/production/uploads/6578459d62d3ac1817ed79fe/AXDJuwLUoEOb4Fj3U0Xxo.jpeg
Shaofei Cai
user
phython96
65519bce32f278f503b33701
Anji Liu
claimed_verified
2024-10-28T15:16:28.982Z
66757955b3882fd587d5f363
/avatars/41fdfa19057ad3ea5ece29b94f163218.svg
Anji Liu
user
anjiliu
65519bce32f278f503b33702
Yonggang Jin
admin_assigned
2023-11-13T09:10:11.688Z
648d9106941869a01cfb526e
https://cdn-avatars.huggingface.co/v1/production/uploads/648d9106941869a01cfb526e/wt1PWbfEuA5cU_blssLWZ.jpeg
Yonggang Jin
user
ygjin11
65519bce32f278f503b33703
Jinbing Hou
65519bce32f278f503b33704
Bowei Zhang
65519bce32f278f503b33705
Haowei Lin
claimed_verified
2025-04-08T07:03:26.384Z
63453f02a05b51f7ded3c579
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/u7p-wTj8zyeWpSFRI8Go0.png
Andy Lin
user
pkuHaowei
65519bce32f278f503b33706
Zhaofeng He
65519bce32f278f503b33707
Zilong Zheng
admin_assigned
2023-11-13T09:11:24.628Z
63a95a6a7930fa8c7dd63d4e
/avatars/d9d0420f7ddfe2f3a7e029fb05f1c89f.svg
Zilong Zheng
user
zlzheng
65519bce32f278f503b33708
Yaodong Yang
65519bce32f278f503b33709
Xiaojian Ma
admin_assigned
2023-11-13T09:12:01.241Z
60dd0e36a15ddd7d2006d2e9
/avatars/8bd98177a79efbf295be8f6457683297.svg
Xiaojian Ma
user
jeasinema
65519bce32f278f503b3370a
Yitao Liang
admin_assigned
2023-11-13T09:12:10.164Z
64683a5776bb704aa14588b7
/avatars/e532756f52c5b95981470ace41a10556.svg
Yitao Liang
user
YitaoLiang
JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal
  Language Models
Achieving human-like planning and control with multimodal observations in an
open world is a key milestone for more functional generalist agents. Existing
approaches can handle certain long-horizon tasks in an open world. However,
they still struggle when the number of open-world tasks could potentially be
infinite and lack the capability to progressively enhance task completion as
game time progresses. We introduce JARVIS-1, an open-world agent that can
perceive multimodal input (visual observations and human instructions),
generate sophisticated plans, and perform embodied control, all within the
popular yet challenging open-world Minecraft universe. Specifically, we develop
JARVIS-1 on top of pre-trained multimodal language models, which map visual
observations and textual instructions to plans. The plans will be ultimately
dispatched to the goal-conditioned controllers. We outfit JARVIS-1 with a
multimodal memory, which facilitates planning using both pre-trained knowledge
and its actual game survival experiences. In our experiments, JARVIS-1 exhibits
nearly perfect performances across over 200 varying tasks from the Minecraft
Universe Benchmark, ranging from entry to intermediate levels. JARVIS-1 has
achieved a completion rate of 12.5% in the long-horizon diamond pickaxe task.
This represents a significant increase up to 5 times compared to previous
records. Furthermore, we show that JARVIS-1 is able to self-improve
following a life-long learning paradigm thanks to multimodal memory, sparking a
more general intelligence and improved autonomy. The project page is available
at https://craftjarvis-jarvis1.github.io.
JARVIS-1, an open-world agent in Minecraft, uses multimodal perception and a memory system to perform complex tasks and improve over time.
multimodal observations JARVIS-1 pre-trained multimodal language models multimodal memory goal-conditioned controllers Minecraft Universe Benchmark long-horizon diamond pickaxe task self-improve life-long learning general intelligence autonomy