title: Pengi: An Audio Language Model for Audio Tasks
summary: In the domain of audio processing, Transfer Learning has facilitated the rise
  of Self-Supervised Learning and Zero-Shot Learning techniques. These approaches
  have led to the development of versatile models capable of tackling a wide
  array of tasks, while delivering state-of-the-art performance. However, current
  models inherently lack the capacity to produce the requisite language for
  open-ended tasks, such as Audio Captioning or Audio Question & Answering. We
  introduce Pengi, a novel Audio Language Model that leverages Transfer Learning
  by framing all audio tasks as text-generation tasks. It takes as input, an
  audio recording, and text, and generates free-form text as output. The input
  audio is represented as a sequence of continuous embeddings by an audio
  encoder. A text encoder does the same for the corresponding text input. Both
  sequences are combined as a prefix to prompt a pre-trained frozen language
  model. The unified architecture of Pengi enables open-ended tasks and
  close-ended tasks without any additional fine-tuning or task-specific
  extensions. When evaluated on 22 downstream tasks, our approach yields
  state-of-the-art performance in several of them. Our results show that
  connecting language models with audio models is a major step towards
  general-purpose audio understanding
paper_authors._id: 646af7b835c7a57f936d2222
paper_authors.name: Soham Deshmukh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-02T11:35:09.113Z
paper_authors._id: 6541a9d6aa727b1a95b19133
paper_authors.avatarUrl: /avatars/52d4348427c05e4209a85dff417bf8e3.svg
paper_authors.fullname: Soham Deshmukh
paper_authors.type: user
paper_authors.user: soham97
paper_authors._id: 646af7b835c7a57f936d2223
paper_authors.name: Benjamin Elizalde
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-09T07:54:05.376Z
paper_authors._id: 652087323a0f8319475d4e65
paper_authors.avatarUrl: /avatars/d7039f3c210eb3842bc5a1a507a72290.svg
paper_authors.fullname: Benjamin Elizalde
paper_authors.type: user
paper_authors.user: benmael
paper_authors._id: 646af7b835c7a57f936d2224
paper_authors.name: Rita Singh
paper_authors._id: 646af7b835c7a57f936d2225
paper_authors.name: Huaming Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-26T08:49:58.040Z
paper_authors._id: 6214395ab633fd62c872561c
paper_authors.avatarUrl: /avatars/0030db39a509567f3a35d7c19668c512.svg
paper_authors.fullname: huaming.wang
paper_authors.type: user
paper_authors.user: Huaming
paper_title: Pengi: An Audio Language Model for Audio Tasks
paper_summary: In the domain of audio processing, Transfer Learning has facilitated the rise
  of Self-Supervised Learning and Zero-Shot Learning techniques. These approaches
  have led to the development of versatile models capable of tackling a wide
  array of tasks, while delivering state-of-the-art performance. However, current
  models inherently lack the capacity to produce the requisite language for
  open-ended tasks, such as Audio Captioning or Audio Question & Answering. We
  introduce Pengi, a novel Audio Language Model that leverages Transfer Learning
  by framing all audio tasks as text-generation tasks. It takes as input, an
  audio recording, and text, and generates free-form text as output. The input
  audio is represented as a sequence of continuous embeddings by an audio
  encoder. A text encoder does the same for the corresponding text input. Both
  sequences are combined as a prefix to prompt a pre-trained frozen language
  model. The unified architecture of Pengi enables open-ended tasks and
  close-ended tasks without any additional fine-tuning or task-specific
  extensions. When evaluated on 22 downstream tasks, our approach yields
  state-of-the-art performance in several of them. Our results show that
  connecting language models with audio models is a major step towards
  general-purpose audio understanding
paper_ai_summary: Pengi, an audio language model using transfer learning, achieves state-of-the-art performance across various audio tasks by framing them as text-generation tasks.
paper_ai_keywords: Self-Supervised Learning Zero-Shot Learning Transfer Learning Audio Captioning Audio Question & Answeing Audio Language Model continuous embeddings audio encoder text encoder language model open-ended tasks close-ended tasks