3.1 Environment and Action Space
Experimental Environment: Our experimental
environment is built on a command-line interface
(CLI), allowing the agent to interact with smart-
phone apps. We chose the Android operating sys-
tem for our experiments. The agent receives two
key inputs: a real-time screenshot showing the
app’s interface and an XML file detailing the in-
teractive elements. To enhance the agent’s ability
to identify and interact with these elements seam-
lessly, we assign each element a unique identifier.
These identifiers are derived either from the re-
source ID in the XML file (if provided) or are con-
structed by combining the class name, size, and
content of the element. These elements are over-
laid as semi-transparent numbers on the screenshot.

4

This helps the agent to interact accurately without
needing to specify exact positions on the screen
and enhances the agent’s precision in controlling
the phone.
AppAgents