LLM reasoning to decompose the instruction into sub-tasks
or related tasks, which will then be used as textual queries
to retrieve relevant planning experiences as references for
solving the current task. For instance, consider the instruc-
tion "craft 1 enchanting table with empty inventory" as
shown in Figure 5. JARVIS-1 queries the MLMs to identify
the tasks that are required for achieving the main task in
a backward search fashion, e.g., “obtain book
/diamond
with empty inventory”. The search depth
is bounded for efficiency. Further, instead of relying solely
on retrieval based on the text query [Wang et al., 2023b,
Zhu et al., 2023], we also propose to append the agent’s
current visual observation to the textual query, resulting in a
multimodal query to take the situation into account during
memory retrieval.
Multimodal retrieval. After obtaining the textual and vi-
sual query, we compute the alignment between the query
and each trajectory in multimodal memory. We first use
JARVIS-1