SoloAudio: Target Sound Extraction with Language-oriented Audio Diffusion Transformer
<|reference_start|>SoloAudio: Target Sound Extraction with Language-oriented Audio Diffusion Transformer: In this paper, we introduce SoloAudio, a novel diffusion-based generative model for target sound extraction (TSE). Our approach trains latent diffusion models on audio, replacing the previous U-Net backbone with a skip-connected Transformer that operates on latent features. SoloAudio supports both audio-oriented and language-oriented TSE by utilizing a CLAP model as the feature extractor for target sounds. Furthermore, SoloAudio leverages synthetic audio generated by state-of-the-art text-to-audio models for training, demonstrating strong generalization to out-of-domain data and unseen sound events. We evaluate this approach on the FSD Kaggle 2018 mixture dataset and real data from AudioSet, where SoloAudio achieves the state-of-the-art results on both in-domain and out-of-domain data, and exhibits impressive zero-shot and few-shot capabilities. Source code and demos are released.<|reference_end|>
@article{wang2024soloaudio:,
    title={SoloAudio: Target Sound Extraction with Language-oriented Audio
  Diffusion Transformer},
    author={Helin Wang, Jiarui Hai, Yen-Ju Lu, Karan Thakkar, Mounya Elhilali,
  Najim Dehak},
    journal={arXiv preprint arXiv:2409.08425},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.08425},
    primaryClass={eess.AS cs.SD}
}