GazePointAR: A Context-Aware Multimodal Voice Assistant for Pronoun Disambiguation in Wearable Augmented Reality
<|reference_start|>GazePointAR: A Context-Aware Multimodal Voice Assistant for Pronoun Disambiguation in Wearable Augmented Reality: Voice assistants (VAs) like Siri and Alexa are transforming human-computer interaction; however, they lack awareness of users' spatiotemporal context, resulting in limited performance and unnatural dialogue. We introduce GazePointAR, a fully-functional context-aware VA for wearable augmented reality that leverages eye gaze, pointing gestures, and conversation history to disambiguate speech queries. With GazePointAR, users can ask "what's over there?" or "how do I solve this math problem?" simply by looking and/or pointing. We evaluated GazePointAR in a three-part lab study (N=12): (1) comparing GazePointAR to two commercial systems; (2) examining GazePointAR's pronoun disambiguation across three tasks; (3) and an open-ended phase where participants could suggest and try their own context-sensitive queries. Participants appreciated the naturalness and human-like nature of pronoun-driven queries, although sometimes pronoun use was counter-intuitive. We then iterated on GazePointAR and conducted a first-person diary study examining how GazePointAR performs in-the-wild. We conclude by enumerating limitations and design considerations for future context-aware VAs.<|reference_end|>
@article{lee2024gazepointar:,
    title={GazePointAR: A Context-Aware Multimodal Voice Assistant for Pronoun
  Disambiguation in Wearable Augmented Reality},
    author={Jaewook Lee, Jun Wang, Elizabeth Brown, Liam Chu, Sebastian S.
  Rodriguez, Jon E. Froehlich},
    journal={arXiv preprint arXiv:2404.08213},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.08213},
    primaryClass={cs.HC}
}