Pengi: An Audio Language Model for Audio Tasks
In the domain of audio processing, Transfer Learning has facilitated the rise
of Self-Supervised Learning and Zero-Shot Learning techniques. These approaches
have led to the development of versatile models capable of tackling a wide
array of tasks, while delivering state-of-the-art performance. However, current
models inherently lack the capacity to produce the requisite language for
open-ended tasks, such as Audio Captioning or Audio Question & Answering. We
introduce Pengi, a novel Audio Language Model that leverages Transfer Learning
by framing all audio tasks as text-generation tasks. It takes as input, an
audio recording, and text, and generates free-form text as output. The input
audio is represented as a sequence of continuous embeddings by an audio
encoder. A text encoder does the same for the corresponding text input. Both
sequences are combined as a prefix to prompt a pre-trained frozen language
model. The unified architecture of Pengi enables open-ended tasks and
close-ended tasks without any additional fine-tuning or task-specific
extensions. When evaluated on 22 downstream tasks, our approach yields
state-of-the-art performance in several of them. Our results show that
connecting language models with audio models is a major step towards
general-purpose audio understanding
646af7b835c7a57f936d2222
Soham Deshmukh
claimed_verified
2023-11-02T11:35:09.113Z
6541a9d6aa727b1a95b19133
/avatars/52d4348427c05e4209a85dff417bf8e3.svg
Soham Deshmukh
user
soham97
646af7b835c7a57f936d2223
Benjamin Elizalde
claimed_verified
2023-10-09T07:54:05.376Z
652087323a0f8319475d4e65
/avatars/d7039f3c210eb3842bc5a1a507a72290.svg
Benjamin Elizalde
user
benmael
646af7b835c7a57f936d2224
Rita Singh
646af7b835c7a57f936d2225
Huaming Wang
admin_assigned
2023-05-26T08:49:58.040Z
6214395ab633fd62c872561c
/avatars/0030db39a509567f3a35d7c19668c512.svg
huaming.wang
user
Huaming
Pengi: An Audio Language Model for Audio Tasks
In the domain of audio processing, Transfer Learning has facilitated the rise
of Self-Supervised Learning and Zero-Shot Learning techniques. These approaches
have led to the development of versatile models capable of tackling a wide
array of tasks, while delivering state-of-the-art performance. However, current
models inherently lack the capacity to produce the requisite language for
open-ended tasks, such as Audio Captioning or Audio Question & Answering. We
introduce Pengi, a novel Audio Language Model that leverages Transfer Learning
by framing all audio tasks as text-generation tasks. It takes as input, an
audio recording, and text, and generates free-form text as output. The input
audio is represented as a sequence of continuous embeddings by an audio
encoder. A text encoder does the same for the corresponding text input. Both
sequences are combined as a prefix to prompt a pre-trained frozen language
model. The unified architecture of Pengi enables open-ended tasks and
close-ended tasks without any additional fine-tuning or task-specific
extensions. When evaluated on 22 downstream tasks, our approach yields
state-of-the-art performance in several of them. Our results show that
connecting language models with audio models is a major step towards
general-purpose audio understanding
Pengi, an audio language model using transfer learning, achieves state-of-the-art performance across various audio tasks by framing them as text-generation tasks.
Self-Supervised Learning Zero-Shot Learning Transfer Learning Audio Captioning Audio Question & Answeing Audio Language Model continuous embeddings audio encoder text encoder language model open-ended tasks close-ended tasks