title: Universal Source Separation with Weakly Labelled Data
summary: Universal source separation (USS) is a fundamental research task for
  computational auditory scene analysis, which aims to separate mono recordings
  into individual source tracks. There are three potential challenges awaiting
  the solution to the audio source separation task. First, previous audio source
  separation systems mainly focus on separating one or a limited number of
  specific sources. There is a lack of research on building a unified system that
  can separate arbitrary sources via a single model. Second, most previous
  systems require clean source data to train a separator, while clean source data
  are scarce. Third, there is a lack of USS system that can automatically detect
  and separate active sound classes in a hierarchical level. To use large-scale
  weakly labeled/unlabeled audio data for audio source separation, we propose a
  universal audio source separation framework containing: 1) an audio tagging
  model trained on weakly labeled data as a query net; and 2) a conditional
  source separation model that takes query net outputs as conditions to separate
  arbitrary sound sources. We investigate various query nets, source separation
  models, and training strategies and propose a hierarchical USS strategy to
  automatically detect and separate sound classes from the AudioSet ontology. By
  solely leveraging the weakly labelled AudioSet, our USS system is successful in
  separating a wide variety of sound classes, including sound event separation,
  music source separation, and speech enhancement. The USS system achieves an
  average signal-to-distortion ratio improvement (SDRi) of 5.57 dB over 527 sound
  classes of AudioSet; 10.57 dB on the DCASE 2018 Task 2 dataset; 8.12 dB on the
  MUSDB18 dataset; an SDRi of 7.28 dB on the Slakh2100 dataset; and an SSNR of
  9.00 dB on the voicebank-demand dataset. We release the source code at
  https://github.com/bytedance/uss
paper_authors._id: 6461a8792815a070474e21c8
paper_authors.name: Qiuqiang Kong
paper_authors._id: 6461a8792815a070474e21c9
paper_authors.name: Ke Chen
paper_authors._id: 6461a8792815a070474e21ca
paper_authors.name: Haohe Liu
paper_authors._id: 6461a8792815a070474e21cb
paper_authors.name: Xingjian Du
paper_authors._id: 6461a8792815a070474e21cc
paper_authors.name: Taylor Berg-Kirkpatrick
paper_authors._id: 6461a8792815a070474e21cd
paper_authors.name: Shlomo Dubnov
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-05-15T03:56:12.510Z
paper_authors._id: 643da2c837ca9134b589099a
paper_authors.avatarUrl: /avatars/67c317d2b1644b127c4eed53da301ad9.svg
paper_authors.fullname: Shlomo Dubnov
paper_authors.type: user
paper_authors.user: sdubnov
paper_authors._id: 6461a8792815a070474e21ce
paper_authors.name: Mark D. Plumbley
paper_title: Universal Source Separation with Weakly Labelled Data
paper_summary: Universal source separation (USS) is a fundamental research task for
  computational auditory scene analysis, which aims to separate mono recordings
  into individual source tracks. There are three potential challenges awaiting
  the solution to the audio source separation task. First, previous audio source
  separation systems mainly focus on separating one or a limited number of
  specific sources. There is a lack of research on building a unified system that
  can separate arbitrary sources via a single model. Second, most previous
  systems require clean source data to train a separator, while clean source data
  are scarce. Third, there is a lack of USS system that can automatically detect
  and separate active sound classes in a hierarchical level. To use large-scale
  weakly labeled/unlabeled audio data for audio source separation, we propose a
  universal audio source separation framework containing: 1) an audio tagging
  model trained on weakly labeled data as a query net; and 2) a conditional
  source separation model that takes query net outputs as conditions to separate
  arbitrary sound sources. We investigate various query nets, source separation
  models, and training strategies and propose a hierarchical USS strategy to
  automatically detect and separate sound classes from the AudioSet ontology. By
  solely leveraging the weakly labelled AudioSet, our USS system is successful in
  separating a wide variety of sound classes, including sound event separation,
  music source separation, and speech enhancement. The USS system achieves an
  average signal-to-distortion ratio improvement (SDRi) of 5.57 dB over 527 sound
  classes of AudioSet; 10.57 dB on the DCASE 2018 Task 2 dataset; 8.12 dB on the
  MUSDB18 dataset; an SDRi of 7.28 dB on the Slakh2100 dataset; and an SSNR of
  9.00 dB on the voicebank-demand dataset. We release the source code at
  https://github.com/bytedance/uss