title: Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding
    of Sound and Language
summary: We present DenseAV, a novel dual encoder grounding architecture that learns
  high-resolution, semantically meaningful, and audio-visually aligned features
  solely through watching videos. We show that DenseAV can discover the
  ``meaning'' of words and the ``location'' of sounds without explicit
  localization supervision. Furthermore, it automatically discovers and
  distinguishes between these two types of associations without supervision. We
  show that DenseAV's localization abilities arise from a new multi-head feature
  aggregation operator that directly compares dense image and audio
  representations for contrastive learning. In contrast, many other systems that
  learn ``global'' audio and video representations cannot localize words and
  sound. Finally, we contribute two new datasets to improve the evaluation of AV
  representations through speech and sound prompted semantic segmentation. On
  these and other datasets we show DenseAV dramatically outperforms the prior art
  on speech and sound prompted semantic segmentation. DenseAV outperforms the
  previous state-of-the-art, ImageBind, on cross-modal retrieval using fewer than
  half of the parameters. Project Page:
  https://aka.ms/denseav{https://aka.ms/denseav}
paper_authors._id: 6669c2e9832c8542889e1e20
paper_authors.name: Mark Hamilton
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-13T02:38:29.208Z
paper_authors._id: 62dae3734398e21bf7f53443
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1658512237806-noauth.jpeg
paper_authors.fullname: Mark Hamilton
paper_authors.type: user
paper_authors.user: mhamilton723
paper_authors._id: 6669c2e9832c8542889e1e21
paper_authors.name: Andrew Zisserman
paper_authors._id: 6669c2e9832c8542889e1e22
paper_authors.name: John R. Hershey
paper_authors._id: 6669c2e9832c8542889e1e23
paper_authors.name: William T. Freeman
paper_title: Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding
    of Sound and Language
paper_summary: We present DenseAV, a novel dual encoder grounding architecture that learns
  high-resolution, semantically meaningful, and audio-visually aligned features
  solely through watching videos. We show that DenseAV can discover the
  ``meaning'' of words and the ``location'' of sounds without explicit
  localization supervision. Furthermore, it automatically discovers and
  distinguishes between these two types of associations without supervision. We
  show that DenseAV's localization abilities arise from a new multi-head feature
  aggregation operator that directly compares dense image and audio
  representations for contrastive learning. In contrast, many other systems that
  learn ``global'' audio and video representations cannot localize words and
  sound. Finally, we contribute two new datasets to improve the evaluation of AV
  representations through speech and sound prompted semantic segmentation. On
  these and other datasets we show DenseAV dramatically outperforms the prior art
  on speech and sound prompted semantic segmentation. DenseAV outperforms the
  previous state-of-the-art, ImageBind, on cross-modal retrieval using fewer than
  half of the parameters. Project Page:
  https://aka.ms/denseav{https://aka.ms/denseav}
paper_ai_summary: DenseAV, a dual encoder model, learns audio-visual features without supervision, excelling in semantic segmentation and cross-modal retrieval with minimal parameters.
paper_ai_keywords: dual encoder high-resolution semantically meaningful audio-visually aligned multi-head feature aggregation contrastive learning speech prompted semantic segmentation sound prompted semantic segmentation cross-modal retrieval ImageBind