where x, y, and z denote instruction, plans, and retrieved
memory entries respectively, and pη and pθ are denoted as re-
trieval and planning models. Such retrieval-augmented plan-
ning method helps JARVIS-1 ground the internal knowl-
edge into the open-ended environments efficiently and lever-
age the historical interaction feedback to solve the halluci-
nation within LLMs and produce more accurate plans.
Multimodal memory. We have demonstrated the layout of
our multimodal memory on the right side of Figure 5. From
a high level, it is a key-value memory where the keys are
multimodal, comprising both the task and the observation
(or situation) made when this memory entry was created.
The values are the plans that were successfully executed.
Note that, since the plans in an open-world environment
like Minecraft are situated (see Section 2.1), there could
be multiple entries that are with the same task but different
observations and plans. As a result, JARVIS-1 needs to
JARVIS-1