ing the relevant memory entries, the planning skill of our
MLM-based agent can be strengthened from the agent’s own
interactions with the environment in an in-context manner.
Finally, JARVIS-1 is able to evolve throughout the game-
play by proposing tasks on its own (i.e. self-instruct) as a
means of exploration and saving the obtained experiences
in the multimodal memory, therefore facilitating better rea-
soning and planning. This self-improving ability sparks its
potential for a higher level of autonomy.
Our main evaluations are conducted in Minecraft, with more
than 200 tasks selected from the Minecraft Universe Bench-
mark [Lin et al., 2023a], with no demonstration provided.
The tasks cover a broad spectrum from the early game (e.g.
JARVIS-1