title: ReCLAP: Improving Zero Shot Audio Classification by Describing Sounds
summary: Open-vocabulary audio-language models, like CLAP, offer a promising approach
  for zero-shot audio classification (ZSAC) by enabling classification with any
  arbitrary set of categories specified with natural language prompts. In this
  paper, we propose a simple but effective method to improve ZSAC with CLAP.
  Specifically, we shift from the conventional method of using prompts with
  abstract category labels (e.g., Sound of an organ) to prompts that describe
  sounds using their inherent descriptive features in a diverse context (e.g.,The
  organ's deep and resonant tones filled the cathedral.). To achieve this, we
  first propose ReCLAP, a CLAP model trained with rewritten audio captions for
  improved understanding of sounds in the wild. These rewritten captions describe
  each sound event in the original caption using their unique discriminative
  characteristics. ReCLAP outperforms all baselines on both multi-modal
  audio-text retrieval and ZSAC. Next, to improve zero-shot audio classification
  with ReCLAP, we propose prompt augmentation. In contrast to the traditional
  method of employing hand-written template prompts, we generate custom prompts
  for each unique label in the dataset. These custom prompts first describe the
  sound event in the label and then employ them in diverse scenes. Our proposed
  method improves ReCLAP's performance on ZSAC by 1%-18% and outperforms all
  baselines by 1% - 55%.
paper_authors._id: 66e8ee1ee39084cfc7f233c9
paper_authors.name: Sreyan Ghosh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T15:36:37.734Z
paper_authors._id: 62c9664eb34e600d7eaa4beb
paper_authors.avatarUrl: /avatars/ca23ecdec2d31c99ecce97d9b180ae0c.svg
paper_authors.fullname: Ghosh
paper_authors.type: user
paper_authors.user: Sreyan88
paper_authors._id: 66e8ee1ee39084cfc7f233ca
paper_authors.name: Sonal Kumar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-17T15:08:14.641Z
paper_authors._id: 65203f31d6dec04046139874
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65203f31d6dec04046139874/1LDWW4KblYeIDve9F_kUA.png
paper_authors.fullname: Sonal Kumar
paper_authors.type: user
paper_authors.user: sonalkum
paper_authors._id: 66e8ee1ee39084cfc7f233cb
paper_authors.name: Chandra Kiran Reddy Evuru
paper_authors._id: 66e8ee1ee39084cfc7f233cc
paper_authors.name: Oriol Nieto
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-17T07:50:57.540Z
paper_authors._id: 657a29d7d3e458405b8dc146
paper_authors.avatarUrl: /avatars/1fe425dcf3133462fb6376e8e1550b4a.svg
paper_authors.fullname: Oriol Nieto
paper_authors.type: user
paper_authors.user: urinieto
paper_authors._id: 66e8ee1ee39084cfc7f233cd
paper_authors.name: Ramani Duraiswami
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-17T07:50:51.405Z
paper_authors._id: 65b8f0339b7250e205100190
paper_authors.avatarUrl: /avatars/29df91154c5911a44ef503bb3580ac81.svg
paper_authors.fullname: Ramani Duraiswami
paper_authors.type: user
paper_authors.user: RamaniD
paper_authors._id: 66e8ee1ee39084cfc7f233ce
paper_authors.name: Dinesh Manocha
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-17T07:50:44.680Z
paper_authors._id: 6537a569568d8be8fa096b8c
paper_authors.avatarUrl: /avatars/bfda5cb252d8b5bc3ad737d99c0d7f49.svg
paper_authors.fullname: Dinesh Manocha
paper_authors.type: user
paper_authors.user: manocha
paper_title: ReCLAP: Improving Zero Shot Audio Classification by Describing Sounds
paper_summary: Open-vocabulary audio-language models, like CLAP, offer a promising approach
  for zero-shot audio classification (ZSAC) by enabling classification with any
  arbitrary set of categories specified with natural language prompts. In this
  paper, we propose a simple but effective method to improve ZSAC with CLAP.
  Specifically, we shift from the conventional method of using prompts with
  abstract category labels (e.g., Sound of an organ) to prompts that describe
  sounds using their inherent descriptive features in a diverse context (e.g.,The
  organ's deep and resonant tones filled the cathedral.). To achieve this, we
  first propose ReCLAP, a CLAP model trained with rewritten audio captions for
  improved understanding of sounds in the wild. These rewritten captions describe
  each sound event in the original caption using their unique discriminative
  characteristics. ReCLAP outperforms all baselines on both multi-modal
  audio-text retrieval and ZSAC. Next, to improve zero-shot audio classification
  with ReCLAP, we propose prompt augmentation. In contrast to the traditional
  method of employing hand-written template prompts, we generate custom prompts
  for each unique label in the dataset. These custom prompts first describe the
  sound event in the label and then employ them in diverse scenes. Our proposed
  method improves ReCLAP's performance on ZSAC by 1%-18% and outperforms all
  baselines by 1% - 55%.
paper_ai_summary: A method employing ReCLAP and prompt augmentation enhances zero-shot audio classification by using descriptive audio features in context-specific prompts.
paper_ai_keywords: CLAP zero-shot audio classification ZSAC ReCLAP multi-modal audio-text retrieval prompt augmentation