  We propose a self-supervised approach for learning to perform audio source
separation in videos based on natural language queries, using only unlabeled
video and audio pairs as training data. A key challenge in this task is
learning to associate the linguistic description of a sound-emitting object to
its visual features and the corresponding components of the audio waveform, all
without access to annotations during training. To overcome this challenge, we
adapt off-the-shelf vision-language foundation models to provide pseudo-target
supervision via two novel loss functions and encourage a stronger alignment
between the audio, visual and natural language modalities. During inference,
our approach can separate sounds given text, video and audio input, or given
text and audio input alone. We demonstrate the effectiveness of our
self-supervised approach on three audio-visual separation datasets, including
MUSIC, SOLOS and AudioSet, where we outperform state-of-the-art strongly
supervised approaches despite not using object detectors or text labels during
training.

Computer Vision and Pattern Recognition;Artificial Intelligence;Computation and Language