  Dynamically synthesizing talking speech that actively responds to a listening
head is critical during the face-to-face interaction. For example, the speaker
could take advantage of the listener's facial expression to adjust the tones,
stressed syllables, or pauses. In this work, we present a new visual-aware
text-to-speech (VA-TTS) task to synthesize speech conditioned on both textual
inputs and sequential visual feedback (e.g., nod, smile) of the listener in
face-to-face communication. Different from traditional text-to-speech, VA-TTS
highlights the impact of visual modality. On this newly-minted task, we devise
a baseline model to fuse phoneme linguistic information and listener visual
signals for speech synthesis. Extensive experiments on multimodal conversation
dataset ViCo-X verify our proposal for generating more natural audio with
scenario-appropriate rhythm and prosody.

Audio and Speech Processing;Computation and Language;Sound