18

JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models

mouse movements, as with human gameplay. As with human gameplay, when in-game GUIs are not open, mouse X and Y
actions change the agent’s yaw and pitch, respectively. When a GUI is open, camera actions move the mouse cursor. Mouse
movements are relative (i.e. they move the mouse or camera relative to the current position, and thus their effect depends on
the current position). In Minecraft, interacting with the inventory requires precise mouse movements for tasks like crafting
and smelting, while mining and navigating the world can be accomplished with broader mouse actions. To be enable to
achieve both the same action space, we abstract the craft and smelt action with GUI into functional binary actions, which
are same as MineDojo [Fan et al., 2022]. The detailed action space are described in Table 6.

Index

Human Action Description

Table 6: The action space we use in Minecraft.
JARVIS-1