DeFT-Mamba: Universal Multichannel Sound Separation and Polyphonic Audio Classification
<|reference_start|>DeFT-Mamba: Universal Multichannel Sound Separation and Polyphonic Audio Classification: This paper presents a framework for universal sound separation and polyphonic audio classification, addressing the challenges of separating and classifying individual sound sources in a multichannel mixture. The proposed framework, DeFT-Mamba, utilizes the dense frequency-time attentive network (DeFTAN) combined with Mamba to extract sound objects, capturing the local time-frequency relations through gated convolution block and the global time-frequency relations through position-wise Hybrid Mamba. DeFT-Mamba surpasses existing separation and classification networks by a large margin, particularly in complex scenarios involving in-class polyphony. Additionally, a classification-based source counting method is introduced to identify the presence of multiple sources, outperforming conventional threshold-based approaches. Separation refinement tuning is also proposed to improve performance further. The proposed framework is trained and tested on a multichannel universal sound separation dataset developed in this work, designed to mimic realistic environments with moving sources and varying onsets and offsets of polyphonic events.<|reference_end|>
@article{lee2024deft-mamba:,
    title={DeFT-Mamba: Universal Multichannel Sound Separation and Polyphonic Audio
  Classification},
    author={Dongheon Lee, and Jung-Woo Choi},
    journal={arXiv preprint arXiv:2409.12413},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.12413},
    primaryClass={eess.AS cs.SD}
}