3 Method

This section details the methodology behind our in-
novative multimodal agent framework. This frame-
work enables an agent to interact with smartphone
applications in a manner akin to human behavior.
We first describe the experimental environment and
action space, which are foundational elements of
our system. Next, we discuss the exploration phase,
where the agent learns app functionalities either
through autonomous interactions or by observing
human demonstrations. Finally, we outline the de-
ployment phase, explaining how the agent applies
its acquired knowledge to execute high-level tasks.
AppAgents