  VoxML is a modeling language used to map natural language expressions into
real-time visualizations using commonsense semantic knowledge of objects and
events. Its utility has been demonstrated in embodied simulation environments
and in agent-object interactions in situated multimodal human-agent
collaboration and communication. It introduces the notion of object affordance
(both Gibsonian and Telic) from HRI and robotics, as well as the concept of
habitat (an object's context of use) for interactions between a rational agent
and an object. This paper aims to specify VoxML as an annotation language in
general abstract terms. It then shows how it works on annotating linguistic
data that express visually perceptible human-object interactions. The
annotation structures thus generated will be interpreted against the enriched
minimal model created by VoxML as a modeling language while supporting the
modeling purposes of VoxML linguistically.

Computation and Language