Synthio: Augmenting Small-Scale Audio Classification Datasets with
  Synthetic Data
We present Synthio, a novel approach for augmenting small-scale audio
classification datasets with synthetic data. Our goal is to improve audio
classification accuracy with limited labeled data. Traditional data
augmentation techniques, which apply artificial transformations (e.g., adding
random noise or masking segments), struggle to create data that captures the
true diversity present in real-world audios. To address this shortcoming, we
propose to augment the dataset with synthetic audio generated from
text-to-audio (T2A) diffusion models. However, synthesizing effective
augmentations is challenging because not only should the generated data be
acoustically consistent with the underlying small-scale dataset, but they
should also have sufficient compositional diversity. To overcome the first
challenge, we align the generations of the T2A model with the small-scale
dataset using preference optimization. This ensures that the acoustic
characteristics of the generated data remain consistent with the small-scale
dataset. To address the second challenge, we propose a novel caption generation
technique that leverages the reasoning capabilities of Large Language Models to
(1) generate diverse and meaningful audio captions and (2) iteratively refine
their quality. The generated captions are then used to prompt the aligned T2A
model. We extensively evaluate Synthio on ten datasets and four simulated
limited-data settings. Results indicate our method consistently outperforms all
baselines by 0.1%-39% using a T2A model trained only on weakly-captioned
AudioSet.
66ff54d92f0dc1b2315c7bb0
Sreyan Ghosh
admin_assigned
2024-10-04T09:33:46.374Z
627a354cc488a8ce15a2dec5
/avatars/0d99a2fea8b193993fe5b9b7e5b74f40.svg
Sreyan Ghosh
user
SreyanG-NVIDIA
66ff54d92f0dc1b2315c7bb1
Sonal Kumar
claimed_verified
2024-10-28T15:40:35.207Z
65203f31d6dec04046139874
https://cdn-avatars.huggingface.co/v1/production/uploads/65203f31d6dec04046139874/1LDWW4KblYeIDve9F_kUA.png
Sonal Kumar
user
sonalkum
66ff54d92f0dc1b2315c7bb2
Zhifeng Kong
admin_assigned
2024-10-04T09:34:03.072Z
652a4dfc36f031c5e6f8b8a6
/avatars/9fb56b025dc25f91ca6c31136eaf74b2.svg
Zhifeng Kong
user
ZhifengKong
66ff54d92f0dc1b2315c7bb3
Rafael Valle
admin_assigned
2024-10-04T09:34:19.799Z
6440ddd65d600fb09518daa8
/avatars/ac5898afd2082d230e2ebf6fb867ad4f.svg
Rafael Valle
user
rafaelvalle
66ff54d92f0dc1b2315c7bb4
Bryan Catanzaro
admin_assigned
2024-10-04T09:34:25.451Z
6311021788942700629e6247
/avatars/e7adc1632b76e80e7e4a590033d1c20a.svg
Bryan Catanzaro
user
ctnzr
66ff54d92f0dc1b2315c7bb5
Dinesh Manocha
admin_assigned
2024-10-04T09:34:32.757Z
6537a569568d8be8fa096b8c
/avatars/bfda5cb252d8b5bc3ad737d99c0d7f49.svg
Dinesh Manocha
user
manocha
Synthio: Augmenting Small-Scale Audio Classification Datasets with
  Synthetic Data
We present Synthio, a novel approach for augmenting small-scale audio
classification datasets with synthetic data. Our goal is to improve audio
classification accuracy with limited labeled data. Traditional data
augmentation techniques, which apply artificial transformations (e.g., adding
random noise or masking segments), struggle to create data that captures the
true diversity present in real-world audios. To address this shortcoming, we
propose to augment the dataset with synthetic audio generated from
text-to-audio (T2A) diffusion models. However, synthesizing effective
augmentations is challenging because not only should the generated data be
acoustically consistent with the underlying small-scale dataset, but they
should also have sufficient compositional diversity. To overcome the first
challenge, we align the generations of the T2A model with the small-scale
dataset using preference optimization. This ensures that the acoustic
characteristics of the generated data remain consistent with the small-scale
dataset. To address the second challenge, we propose a novel caption generation
technique that leverages the reasoning capabilities of Large Language Models to
(1) generate diverse and meaningful audio captions and (2) iteratively refine
their quality. The generated captions are then used to prompt the aligned T2A
model. We extensively evaluate Synthio on ten datasets and four simulated
limited-data settings. Results indicate our method consistently outperforms all
baselines by 0.1%-39% using a T2A model trained only on weakly-captioned
AudioSet.
Syncio enhances audio classification by augmenting small datasets with合成 audio from text-to-audio diffusion models, ensuring acoustic consistency and compositional diversity through preference optimization and refined caption generation.
text-to-audio diffusion models preference optimization caption generation Large Language Models AudioSet