B.2. Action Space

We design a hybrid action space. Some are directly available to human players, including keypresses, mouse movements,
and clicks, which are similar to VPT [Baker et al., 2022]. The keypresses and clicks are binary functional actions, including
forward, jump, use and attack etc. In addition to the binary (on/off) keypress actions, our action space also includes

18

JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models
JARVIS-1