Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding
  of Sound and Language
We present DenseAV, a novel dual encoder grounding architecture that learns
high-resolution, semantically meaningful, and audio-visually aligned features
solely through watching videos. We show that DenseAV can discover the
``meaning'' of words and the ``location'' of sounds without explicit
localization supervision. Furthermore, it automatically discovers and
distinguishes between these two types of associations without supervision. We
show that DenseAV's localization abilities arise from a new multi-head feature
aggregation operator that directly compares dense image and audio
representations for contrastive learning. In contrast, many other systems that
learn ``global'' audio and video representations cannot localize words and
sound. Finally, we contribute two new datasets to improve the evaluation of AV
representations through speech and sound prompted semantic segmentation. On
these and other datasets we show DenseAV dramatically outperforms the prior art
on speech and sound prompted semantic segmentation. DenseAV outperforms the
previous state-of-the-art, ImageBind, on cross-modal retrieval using fewer than
half of the parameters. Project Page:
https://aka.ms/denseav{https://aka.ms/denseav}
6669c2e9832c8542889e1e20
Mark Hamilton
extracted_confirmed
2024-06-13T02:38:29.208Z
62dae3734398e21bf7f53443
https://cdn-avatars.huggingface.co/v1/production/uploads/1658512237806-noauth.jpeg
Mark Hamilton
user
mhamilton723
6669c2e9832c8542889e1e21
Andrew Zisserman
6669c2e9832c8542889e1e22
John R. Hershey
6669c2e9832c8542889e1e23
William T. Freeman
Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding
  of Sound and Language
We present DenseAV, a novel dual encoder grounding architecture that learns
high-resolution, semantically meaningful, and audio-visually aligned features
solely through watching videos. We show that DenseAV can discover the
``meaning'' of words and the ``location'' of sounds without explicit
localization supervision. Furthermore, it automatically discovers and
distinguishes between these two types of associations without supervision. We
show that DenseAV's localization abilities arise from a new multi-head feature
aggregation operator that directly compares dense image and audio
representations for contrastive learning. In contrast, many other systems that
learn ``global'' audio and video representations cannot localize words and
sound. Finally, we contribute two new datasets to improve the evaluation of AV
representations through speech and sound prompted semantic segmentation. On
these and other datasets we show DenseAV dramatically outperforms the prior art
on speech and sound prompted semantic segmentation. DenseAV outperforms the
previous state-of-the-art, ImageBind, on cross-modal retrieval using fewer than
half of the parameters. Project Page:
https://aka.ms/denseav{https://aka.ms/denseav}
DenseAV, a dual encoder model, learns audio-visual features without supervision, excelling in semantic segmentation and cross-modal retrieval with minimal parameters.
dual encoder high-resolution semantically meaningful audio-visually aligned multi-head feature aggregation contrastive learning speech prompted semantic segmentation sound prompted semantic segmentation cross-modal retrieval ImageBind