title: Synthio: Augmenting Small-Scale Audio Classification Datasets with
    Synthetic Data
summary: We present Synthio, a novel approach for augmenting small-scale audio
  classification datasets with synthetic data. Our goal is to improve audio
  classification accuracy with limited labeled data. Traditional data
  augmentation techniques, which apply artificial transformations (e.g., adding
  random noise or masking segments), struggle to create data that captures the
  true diversity present in real-world audios. To address this shortcoming, we
  propose to augment the dataset with synthetic audio generated from
  text-to-audio (T2A) diffusion models. However, synthesizing effective
  augmentations is challenging because not only should the generated data be
  acoustically consistent with the underlying small-scale dataset, but they
  should also have sufficient compositional diversity. To overcome the first
  challenge, we align the generations of the T2A model with the small-scale
  dataset using preference optimization. This ensures that the acoustic
  characteristics of the generated data remain consistent with the small-scale
  dataset. To address the second challenge, we propose a novel caption generation
  technique that leverages the reasoning capabilities of Large Language Models to
  (1) generate diverse and meaningful audio captions and (2) iteratively refine
  their quality. The generated captions are then used to prompt the aligned T2A
  model. We extensively evaluate Synthio on ten datasets and four simulated
  limited-data settings. Results indicate our method consistently outperforms all
  baselines by 0.1%-39% using a T2A model trained only on weakly-captioned
  AudioSet.
paper_authors._id: 66ff54d92f0dc1b2315c7bb0
paper_authors.name: Sreyan Ghosh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T09:33:46.374Z
paper_authors._id: 627a354cc488a8ce15a2dec5
paper_authors.avatarUrl: /avatars/0d99a2fea8b193993fe5b9b7e5b74f40.svg
paper_authors.fullname: Sreyan Ghosh
paper_authors.type: user
paper_authors.user: SreyanG-NVIDIA
paper_authors._id: 66ff54d92f0dc1b2315c7bb1
paper_authors.name: Sonal Kumar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T15:40:35.207Z
paper_authors._id: 65203f31d6dec04046139874
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65203f31d6dec04046139874/1LDWW4KblYeIDve9F_kUA.png
paper_authors.fullname: Sonal Kumar
paper_authors.type: user
paper_authors.user: sonalkum
paper_authors._id: 66ff54d92f0dc1b2315c7bb2
paper_authors.name: Zhifeng Kong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T09:34:03.072Z
paper_authors._id: 652a4dfc36f031c5e6f8b8a6
paper_authors.avatarUrl: /avatars/9fb56b025dc25f91ca6c31136eaf74b2.svg
paper_authors.fullname: Zhifeng Kong
paper_authors.type: user
paper_authors.user: ZhifengKong
paper_authors._id: 66ff54d92f0dc1b2315c7bb3
paper_authors.name: Rafael Valle
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T09:34:19.799Z
paper_authors._id: 6440ddd65d600fb09518daa8
paper_authors.avatarUrl: /avatars/ac5898afd2082d230e2ebf6fb867ad4f.svg
paper_authors.fullname: Rafael Valle
paper_authors.type: user
paper_authors.user: rafaelvalle
paper_authors._id: 66ff54d92f0dc1b2315c7bb4
paper_authors.name: Bryan Catanzaro
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T09:34:25.451Z
paper_authors._id: 6311021788942700629e6247
paper_authors.avatarUrl: /avatars/e7adc1632b76e80e7e4a590033d1c20a.svg
paper_authors.fullname: Bryan Catanzaro
paper_authors.type: user
paper_authors.user: ctnzr
paper_authors._id: 66ff54d92f0dc1b2315c7bb5
paper_authors.name: Dinesh Manocha
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T09:34:32.757Z
paper_authors._id: 6537a569568d8be8fa096b8c
paper_authors.avatarUrl: /avatars/bfda5cb252d8b5bc3ad737d99c0d7f49.svg
paper_authors.fullname: Dinesh Manocha
paper_authors.type: user
paper_authors.user: manocha
paper_title: Synthio: Augmenting Small-Scale Audio Classification Datasets with
    Synthetic Data
paper_summary: We present Synthio, a novel approach for augmenting small-scale audio
  classification datasets with synthetic data. Our goal is to improve audio
  classification accuracy with limited labeled data. Traditional data
  augmentation techniques, which apply artificial transformations (e.g., adding
  random noise or masking segments), struggle to create data that captures the
  true diversity present in real-world audios. To address this shortcoming, we
  propose to augment the dataset with synthetic audio generated from
  text-to-audio (T2A) diffusion models. However, synthesizing effective
  augmentations is challenging because not only should the generated data be
  acoustically consistent with the underlying small-scale dataset, but they
  should also have sufficient compositional diversity. To overcome the first
  challenge, we align the generations of the T2A model with the small-scale
  dataset using preference optimization. This ensures that the acoustic
  characteristics of the generated data remain consistent with the small-scale
  dataset. To address the second challenge, we propose a novel caption generation
  technique that leverages the reasoning capabilities of Large Language Models to
  (1) generate diverse and meaningful audio captions and (2) iteratively refine
  their quality. The generated captions are then used to prompt the aligned T2A
  model. We extensively evaluate Synthio on ten datasets and four simulated
  limited-data settings. Results indicate our method consistently outperforms all
  baselines by 0.1%-39% using a T2A model trained only on weakly-captioned
  AudioSet.
paper_ai_keywords: Feder approach text-to-audio (T2A) diffusion models preference optimization Large Language Models caption generation acoustically consistent compositional diversity AudioSet