title: SoloAudio: Target Sound Extraction with Language-oriented Audio
    Diffusion Transformer
summary: In this paper, we introduce SoloAudio, a novel diffusion-based generative
  model for target sound extraction (TSE). Our approach trains latent diffusion
  models on audio, replacing the previous U-Net backbone with a skip-connected
  Transformer that operates on latent features. SoloAudio supports both
  audio-oriented and language-oriented TSE by utilizing a CLAP model as the
  feature extractor for target sounds. Furthermore, SoloAudio leverages synthetic
  audio generated by state-of-the-art text-to-audio models for training,
  demonstrating strong generalization to out-of-domain data and unseen sound
  events. We evaluate this approach on the FSD Kaggle 2018 mixture dataset and
  real data from AudioSet, where SoloAudio achieves the state-of-the-art results
  on both in-domain and out-of-domain data, and exhibits impressive zero-shot and
  few-shot capabilities. Source code and demos are released.
paper_authors._id: 66eae4cfca4f15856bb1ceb4
paper_authors.name: Helin Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-19T13:26:27.158Z
paper_authors._id: 63ecfb5ec5b3c734085db9ed
paper_authors.avatarUrl: /avatars/0b1d03dcd7997ad1daa764fb76f88993.svg
paper_authors.fullname: Helin Wang
paper_authors.type: user
paper_authors.user: westbrook
paper_authors._id: 66eae4cfca4f15856bb1ceb5
paper_authors.name: Jiarui Hai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-22T12:48:01.790Z
paper_authors._id: 6363e9e4969bdae89e10b5ac
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6363e9e4969bdae89e10b5ac/uR0jz8pBJIQPmKC90OJgI.jpeg
paper_authors.fullname: Jiarui Hai
paper_authors.type: user
paper_authors.user: Higobeatz
paper_authors._id: 66eae4cfca4f15856bb1ceb6
paper_authors.name: Yen-Ju Lu
paper_authors._id: 66eae4cfca4f15856bb1ceb7
paper_authors.name: Karan Thakkar
paper_authors._id: 66eae4cfca4f15856bb1ceb8
paper_authors.name: Mounya Elhilali
paper_authors._id: 66eae4cfca4f15856bb1ceb9
paper_authors.name: Najim Dehak
paper_title: SoloAudio: Target Sound Extraction with Language-oriented Audio
    Diffusion Transformer
paper_summary: In this paper, we introduce SoloAudio, a novel diffusion-based generative
  model for target sound extraction (TSE). Our approach trains latent diffusion
  models on audio, replacing the previous U-Net backbone with a skip-connected
  Transformer that operates on latent features. SoloAudio supports both
  audio-oriented and language-oriented TSE by utilizing a CLAP model as the
  feature extractor for target sounds. Furthermore, SoloAudio leverages synthetic
  audio generated by state-of-the-art text-to-audio models for training,
  demonstrating strong generalization to out-of-domain data and unseen sound
  events. We evaluate this approach on the FSD Kaggle 2018 mixture dataset and
  real data from AudioSet, where SoloAudio achieves the state-of-the-art results
  on both in-domain and out-of-domain data, and exhibits impressive zero-shot and
  few-shot capabilities. Source code and demos are released.
paper_ai_keywords: diffusion-based generative model target sound extraction (TSE) latent diffusion models skip-connected Transformer latent features CLAP model feature extractor synthetic audio text-to-audio models generalization out-of-domain data unseen sound events FSD Kaggle 2018 mixture dataset AudioSet state-of-the-art results zero-shot capabilities few-shot capabilities