Lip Interact: Improving Mobile Device Interaction with Silent Speech Commands
We present Lip Interact, an interaction technique that allows users to issue commands on their smartphone through silent speech. Lip Interact repurposes the front camera to capture the user's mouth movements and recognize the issued commands with an end to end deep learning model. Our system supports 44 commands for accessing both system level functionalities (launching apps, changing system settings, and handling pop up windows) and application level functionalities (integrated operations for two apps) We verify the feasibility of Lip Interact with three user experiments: evaluating the recognition accuracy, comparing with touch on input efficiency, and comparing with voiced commands with regards to personal privacy and social norms. We demonstrate that Lip Interact can help users access functionality efficiently in one step, enable one handed input when the other hand is occupied, and assist touch to make interactions more fluent.
UIST