Natural Language Supervision for General-Purpose Audio Representations
Audio-Language models jointly learn multimodal text and audio representations
that enable Zero-Shot inference. Models rely on the encoders to create powerful
representations of the input and generalize to multiple tasks ranging from
sounds, music, and speech. Although models have achieved remarkable
performance, there is still a performance gap with task-specific models. In
this paper, we propose a Contrastive Language-Audio Pretraining model that is
pretrained with a diverse collection of 4.6M audio-text pairs employing two
innovative encoders for Zero-Shot inference. To learn audio representations, we
trained an audio encoder on 22 audio tasks, instead of the standard training of
sound event classification. To learn language representations, we trained an
autoregressive decoder-only model instead of the standard encoder-only models.
Then, the audio and language representations are brought into a joint
multimodal space using Contrastive Learning. We used our encoders to improve
the downstream performance by a margin. We extensively evaluated the
generalization of our representations on 26 downstream tasks, the largest in
the literature. Our model achieves state of the art results in several tasks
leading the way towards general-purpose audio representations.
65010e42868ac1994a3695ee
Benjamin Elizalde
claimed_verified
2023-10-09T07:53:38.171Z
652087323a0f8319475d4e65
/avatars/d7039f3c210eb3842bc5a1a507a72290.svg
Benjamin Elizalde
user
benmael
65010e42868ac1994a3695ef
Soham Deshmukh
claimed_verified
2023-11-02T11:35:06.357Z
6541a9d6aa727b1a95b19133
/avatars/52d4348427c05e4209a85dff417bf8e3.svg
Soham Deshmukh
user
soham97
65010e42868ac1994a3695f0
Huaming Wang
admin_assigned
2023-09-14T07:52:53.485Z
6214395ab633fd62c872561c
/avatars/0030db39a509567f3a35d7c19668c512.svg
huaming.wang
user
Huaming
Natural Language Supervision for General-Purpose Audio Representations
Audio-Language models jointly learn multimodal text and audio representations
that enable Zero-Shot inference. Models rely on the encoders to create powerful
representations of the input and generalize to multiple tasks ranging from
sounds, music, and speech. Although models have achieved remarkable
performance, there is still a performance gap with task-specific models. In
this paper, we propose a Contrastive Language-Audio Pretraining model that is
pretrained with a diverse collection of 4.6M audio-text pairs employing two
innovative encoders for Zero-Shot inference. To learn audio representations, we
trained an audio encoder on 22 audio tasks, instead of the standard training of
sound event classification. To learn language representations, we trained an
autoregressive decoder-only model instead of the standard encoder-only models.
Then, the audio and language representations are brought into a joint
multimodal space using Contrastive Learning. We used our encoders to improve
the downstream performance by a margin. We extensively evaluated the
generalization of our representations on 26 downstream tasks, the largest in
the literature. Our model achieves state of the art results in several tasks
leading the way towards general-purpose audio representations.
A Contrastive Language-Audio Pretraining model using innovative encoders achieves state-of-the-art results in various audio and language tasks through joint multimodal representation learning.
Contrastive Language-Audio Pretraining audio encoder autoregressive decoder-only model Contrastive Learning multimodal space downstream tasks state-of-the-art results general-purpose audio representations