• From LLMs to MLMs. The capability of perceiving
multimodal sensory input is critical to planning in a dy-
namic and open-world world. JARVIS-1 enables this by
chaining a multimodal foundation model together with an
LLM. Compared to LLM “blindly” produces plans, MLM
is able to natively understand the current situation and
plan accordingly. Further, rich environmental feedback

2

.97.97.95.57.96.94.90.92.79.97.75.91.87.91.95.25.25.11.10.04.11.11.04.02.02.05.40.33.32.33.34.09.06.55.05.33.38.30.37.32.36.28.36.28.32.36.38.37.45.50.32.50.31.06.14.05.26.08.18.24.22.23.02.05.05.08.06.02.04.02.05.02.02.14.88.79.84.94.87.90.97.67.90.89.93.92JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models
JARVIS-1