title: Natural Language Supervision for General-Purpose Audio Representations
summary: Audio-Language models jointly learn multimodal text and audio representations
  that enable Zero-Shot inference. Models rely on the encoders to create powerful
  representations of the input and generalize to multiple tasks ranging from
  sounds, music, and speech. Although models have achieved remarkable
  performance, there is still a performance gap with task-specific models. In
  this paper, we propose a Contrastive Language-Audio Pretraining model that is
  pretrained with a diverse collection of 4.6M audio-text pairs employing two
  innovative encoders for Zero-Shot inference. To learn audio representations, we
  trained an audio encoder on 22 audio tasks, instead of the standard training of
  sound event classification. To learn language representations, we trained an
  autoregressive decoder-only model instead of the standard encoder-only models.
  Then, the audio and language representations are brought into a joint
  multimodal space using Contrastive Learning. We used our encoders to improve
  the downstream performance by a margin. We extensively evaluated the
  generalization of our representations on 26 downstream tasks, the largest in
  the literature. Our model achieves state of the art results in several tasks
  leading the way towards general-purpose audio representations.
paper_authors._id: 65010e42868ac1994a3695ee
paper_authors.name: Benjamin Elizalde
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-09T07:53:38.171Z
paper_authors._id: 652087323a0f8319475d4e65
paper_authors.avatarUrl: /avatars/d7039f3c210eb3842bc5a1a507a72290.svg
paper_authors.fullname: Benjamin Elizalde
paper_authors.type: user
paper_authors.user: benmael
paper_authors._id: 65010e42868ac1994a3695ef
paper_authors.name: Soham Deshmukh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-02T11:35:06.357Z
paper_authors._id: 6541a9d6aa727b1a95b19133
paper_authors.avatarUrl: /avatars/52d4348427c05e4209a85dff417bf8e3.svg
paper_authors.fullname: Soham Deshmukh
paper_authors.type: user
paper_authors.user: soham97
paper_authors._id: 65010e42868ac1994a3695f0
paper_authors.name: Huaming Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-14T07:52:53.485Z
paper_authors._id: 6214395ab633fd62c872561c
paper_authors.avatarUrl: /avatars/0030db39a509567f3a35d7c19668c512.svg
paper_authors.fullname: huaming.wang
paper_authors.type: user
paper_authors.user: Huaming
paper_title: Natural Language Supervision for General-Purpose Audio Representations
paper_summary: Audio-Language models jointly learn multimodal text and audio representations
  that enable Zero-Shot inference. Models rely on the encoders to create powerful
  representations of the input and generalize to multiple tasks ranging from
  sounds, music, and speech. Although models have achieved remarkable
  performance, there is still a performance gap with task-specific models. In
  this paper, we propose a Contrastive Language-Audio Pretraining model that is
  pretrained with a diverse collection of 4.6M audio-text pairs employing two
  innovative encoders for Zero-Shot inference. To learn audio representations, we
  trained an audio encoder on 22 audio tasks, instead of the standard training of
  sound event classification. To learn language representations, we trained an
  autoregressive decoder-only model instead of the standard encoder-only models.
  Then, the audio and language representations are brought into a joint
  multimodal space using Contrastive Learning. We used our encoders to improve
  the downstream performance by a margin. We extensively evaluated the
  generalization of our representations on 26 downstream tasks, the largest in
  the literature. Our model achieves state of the art results in several tasks
  leading the way towards general-purpose audio representations.
paper_ai_summary: A Contrastive Language-Audio Pretraining model using innovative encoders achieves state-of-the-art results in various audio and language tasks through joint multimodal representation learning.
paper_ai_keywords: Contrastive Language-Audio Pretraining audio encoder autoregressive decoder-only model Contrastive Learning multimodal space downstream tasks state-of-the-art results general-purpose audio representations