DeFTMamba: Multichannel universal sound separation and polyphonic audio classification

Dongheon Lee¹, and Jung-Woo Choi¹
¹School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea

Abstract: This paper presents a framework for universal sound separation and polyphonic audio classification, addressing the challenges of separating and classifying individual sound sources from a multichannel mixture. The proposed framework, DeFT-Mamba, utilizes the dense frequency-time attentive network (DeFTAN) combined with Mamba to extract sound objects, capturing the local time-frequency relations by gated convolution block and the global time-frequency relations by position-wise Hybrid Mamba. DeFT-Mamba surpasses existing separation and classification networks by a large margin, particularly in complex scenarios with in-class polyphony. Additionally, a classification-based source counting method is introduced to identify the presence of multiple sources, which outperforms conventional threshold-based approaches. A separation refinement tuning is also proposed to improve performance further. The proposed framework is trained and tested on a multichannel universal sound separation dataset developed in this work to mimic realistic environments with moving sources and varying onsets and offsets of polyphonic events.

Dataset explanation (click to expand)