human-aligned actions. To accomplish this, we propose
a multi-modal agent including an interactive planner, a
goal-conditioned controller, and a multimodal memory
of multimodal experiences. Upon receiving a task and the
current observation, JARVIS-1 first utilizes the MLM to
generate a multimodal query (query gen) that retrieves
relevant planning experiences from the memory. These
experiences will then be used along with the planning in-
struction to prompt the MLM-based planner. Leveraging
its own pretrained knowledge as well as the retrieved ref-
erence plans, the planner will ultimately produce a series
of K short-horizon goals g1, . . . , gK to be executed by the
controller. Once the plan is successfully executed, it will
be stored in the memory along with the task and the agent
situation when it was planned. We also empower JARVIS-1
with life-long learning by combining self-instruct, where
JARVIS-1 will propose some tasks for itself to complete as
JARVIS-1