Achieving human-like planning and control with multimodal observations in an open world is a key milestone for
more functional generalist agents. Existing approaches can handle certain long-horizon tasks in an open world.
However, they still struggle when the number of open-world tasks could potentially be infinite and lack the capability
to progressively enhance task completion as game time progresses. We introduce JARVIS-1, an open-world agent
that can perceive multimodal input (visual observations and human instructions), generate sophisticated plans, and
perform embodied control, all within the popular yet challenging open-world Minecraft universe. Specifically, we
develop JARVIS-1 on top of pre-trained multimodal language models, which map visual observations and textual
instructions to plans. The plans will be ultimately dispatched to the goal-conditioned controllers. We outfit JARVIS-1
JARVIS-1