Situation-aware planning with MLM.
To achieve
situation-aware planning, the planner must take the current
observation into account, in addition to the task instruc-
tion [Huang et al., 2022b, Yao et al., 2022]. Specifically, we
begin with translating the multimodal observation into text
descriptions. As opposed to letting the MLM caption the
scene directly, we first extract keywords of Minecraft items
(e.g., "acacia tree", "sheep") from Minecraft wiki and
utilizing GPT [Brown et al., 2020] to generate sentences that
describe these observations. For example, a generated sen-
tence could be "I can see sheep in the acacia plains". Then
the MLM will retrieve the condition sentence according to
current visual observation during planning. Additional situ-
ation details including biome and inventory status are also
converted into text using templates. Finally, we prompt the
MLM again (the language part only) into a plan given the
task instruction and all the aforementioned textual situation
JARVIS-1