Figure 3: Architecture of JARVIS-1 and its self-improving mechanism. (a) JARVIS-1 comprises a memory-augmented multimodal
language model (MLM) that produces plans and a low-level action controller. JARVIS-1 also utilizes a multimodal memory to store and
obtain experiences as references for planning. (b) JARVIS-1 can strengthen its own planning skills through exploration with its own
proposed tasks (self-instruct) and a growing memory that helps with better planning on tasks that has been (partially) visited before.
JARVIS-1