  A speaker extraction algorithm seeks to extract the speech of a target
speaker from a multi-talker speech mixture when given a cue that represents the
target speaker, such as a pre-enrolled speech utterance, or an accompanying
video track. Visual cues are particularly useful when a pre-enrolled speech is
not available. In this work, we don't rely on the target speaker's pre-enrolled
speech, but rather use the target speaker's face track as the speaker cue, that
is referred to as the auxiliary reference, to form an attractor towards the
target speaker. We advocate that the temporal synchronization between the
speech and its accompanying lip movements is a direct and dominant audio-visual
cue. Therefore, we propose a self-supervised pre-training strategy, to exploit
the speech-lip synchronization cue for target speaker extraction, which allows
us to leverage abundant unlabeled in-domain data. We transfer the knowledge
from the pre-trained model to the attractor encoder of the speaker extraction
network. We show that the proposed speaker extraction network outperforms
various competitive baselines in terms of signal quality, perceptual quality,
and intelligibility, achieving state-of-the-art performance.

Audio and Speech Processing