0.95
0.95
0.90
0.30
0.05

0.55
0.50
0.35
0.05
0.00

0.85
0.85
0.75
0.25
0.05

ducted on a subset of Minecraft tasks using different lan-
guage models. Each JARVIS-1 learns for 4 epochs of inter-
action with all task sets and evaluates on task subset across
at least 20 seeds. The experimental results are presented in
Table 3.
Table 3 demonstrates that ChatGPT, despite having fewer
parameters, achieves nearly identical success rates as GPT-4.
This suggests that language models equipped with memory
can significantly enhance planning abilities. In Minecraft-
related tasks, the open-source pre-trained LLaMA2 70B
exhibits a notable performance gap compared to OpenAI
models, particularly in long-horizon tasks. However, by

Table 4: Success rates for memory ablation on Minecraft tasks.
R is reasoning process, T and M represent retrieve with text em-
bedding and multi-modal embedding, respectively.

Memory Retrieval

-
✓
✓
✓

-
T

T+R
M+R

0.85
0.85
0.95
0.94

0.00
0.05
0.25
0.34
JARVIS-1