PHASE-AWARE SPEECH ENHANCEMENT WITH DEEP COMPLEX U-NET
Most deep learning-based models for speech enhancement have mainly focused on estimating the magnitude of spectrogram while reusing the phase from noisy speech for reconstruction. This is due to the difficulty of estimating the phase of clean speech. To improve speech enhancement performance, we tackle the phase estimation problem in three ways. First, we propose Deep Complex U-Net, an advanced U-Net structured model incorporating well-defined complex-valued building blocks to deal with complex-valued spectrograms. Second, we propose a polar coordinate-wise complex-valued masking method to reflect the distribution of complex ideal ratio masks. Third, we define a novel loss function, weighted source-to-distortion ratio (wSDR) loss, which is designed to directly correlate with a quantitative evaluation measure. Our model was evaluated on a mixture of the Voice Bank corpus and DEMAND database, which has been widely used by many deep learning models for speech enhancement. Ablation experiments were conducted on the mixed dataset showing that all three proposed approaches are empirically valid. Experimental results show that the proposed method achieves state-of-the-art performance in all metrics, outperforming previous approaches by a large margin 1 .
This paper tackles one of important speech enhancement issues of how to predict phase information. The authors work on this problem based on three novel techniques, one is to use complex U-net, second is to propose a new complex mask representation, which is well bounded and well model complex mask distribution, and the last is an objective function motivated by SDR. The paper is well written, and also shows the experimental effectiveness of the proposed method by analyzing these three novel techniques and also by comparing the method with other speech enhancement methods. My major concern about this paper is that this paper is a little bit too specific to the speech enhancement applications, which will not be accepted with so many researches in the major ICLR community. My suggestion is to describe some potential applications of this method to the other (speech) applications including speech separation, noise-robust front-end for ASR, TTS, or other speech analysis, and also discuss the possibility of extending this method for multichannel input. I’m more interested in the multichannel enhancement because the phase (difference) is critical in this scenario. 
Comments:
- Introduction: It’s better to cite and discuss the paper of “E. Hakan et al, “Phase-sensitive and recognition-boosted speech separation using deep recurrent neural networks,” Proc. ICASSP’15, pp. 708--712 (2015). This paper is one of the first studies tries to incorporate the phase information to DNN based speech enhancement.
- Several researchers prefer to use LSTM based enhancement method. Please discuss wether this method (objective function and complex masks) can be applied to complex extensions of LSTMs instead of complex U-net.
- Page 2, the first paragraph: You may also refer https://arxiv.org/abs/1810.01395
- Page 3, it’s better to explicitly mention that h = x + i y
- Section 3.3: discuss how we treat STFT/iSTFT operations under a computational graph representation. It is not so obvious.
- Section 3.3: again it’s better to mention E. Hakan’s method here.
- Page 6 footnote: I cannot access to the URL. Please check it.
- Experiments: I think it would be more interesting to add SDR (using speech and noise as a source) to the experimental measure. Some people use SDR as a speech enhancement measure, and I’m expecting that this method can have more reasonable performance since it is optimized based on wSDR.