Toward Natural Gesture/Speech Control of a Large Display
<|reference_start|>Toward Natural Gesture/Speech Control of a Large Display: In recent years because of the advances in computer vision research, free hand gestures have been explored as means of human-computer interaction (HCI). Together with improved speech processing technology it is an important step toward natural multimodal HCI. However, inclusion of non-predefined continuous gestures into a multimodal framework is a challenging problem. In this paper, we propose a structured approach for studying patterns of multimodal language in the context of a 2D-display control. We consider systematic analysis of gestures from observable kinematical primitives to their semantics as pertinent to a linguistic structure. Proposed semantic classification of co-verbal gestures distinguishes six categories based on their spatio-temporal deixis. We discuss evolution of a computational framework for gesture and speech integration which was used to develop an interactive testbed (iMAP). The testbed enabled elicitation of adequate, non-sequential, multimodal patterns in a narrative mode of HCI. Conducted user studies illustrate significance of accounting for the temporal alignment of gesture and speech parts in semantic mapping. Furthermore, co-occurrence analysis of gesture/speech production suggests syntactic organization of gestures at the lexical level.<|reference_end|>
@article{kettebekov2001toward,
    title={Toward Natural Gesture/Speech Control of a Large Display},
    author={S. Kettebekov, R. Sharma},
    journal={arXiv preprint arXiv:cs/0105026},
    year={2001},
    archivePrefix={arXiv},
    eprint={cs/0105026},
    primaryClass={cs.CV cs.HC}
}