summaries: The paper proposes an approach called VoiceFixer which is aimed at restoring degraded speech signals. The paper considers a variety of speech degradations - additive noise, reverberations, clipping and limited bandwidth. The paper describes a two stage approach in which the first stage aims to produce restored mel-spectrogram and then a vocoder is used to synthesize the  speech from the restored mel-spectrogram. Experiments are done using the VCTK dataset and experiments are done using single distortions as well as combinations of all 4 distortions.  This paper proposes a general speech restoration (GSR) task that tries to remove multiple distortions in a single model. In addition, it also presents a generative framework called VoiceFixer consisting of analysis and synthesis stages to address the general speech restoration task. In VoiceFixer, the authors employ a ResNet for modeling the analysis stage and a TFGAN-based neural vocoder for synthesis stage. They report that their baseline GSR and VoiceFixer surpass the single speech restoration (SSR) models with more improved results by the latter. Their idea was well described and the experiments are systematical and extensive. The results are consistent and clear.  The contribution of this paper is to incorporate a variety of speech restoration tasks including speech denoising, super-resolution, dereverberation, declipping, etc. in a single unified task called GSR. Another is the proposal of a well-performing generative speech restoration framework called VoiceFixer.  This paper introduces a unified view of several speech restoration problems including denoising, decliping, dereverberation and audio super-resolution. In order to solve the problem of the general speech restoration task, the authors propose a U-Net architecture which is trained on all of these tasks simultaneously during training time. The authors conduct extensive experiments for the general speech restoration task as well as the individual tasks where they compare the proposed models with more specialized models bounded to each distortion. The experimental results show that the proposed VoiceFixer combination of the model and the analysis-synthesis procedure are capable of effectively removing the speech distortions and in some cases outperform previous approaches in the literature. The paper proposes a single system to deal with the speech enhancement tasks of denoising, dereverb, bandwidth extension (BWE) and declipping. The system is a two-stage system composed of an analysis module producing mel-band masks and a synthesis module using a vocoder. Both modules reuse existing architectures. Results comparing the proposed system to author-derived counterparts of it and to some existing systems for specific tasks show improvement for the proposed system in some cases, while achieving similar performance as existing systems in other cases. Data and code for reproducibility are provided.
abstractText: Speech restoration aims to remove distortions in speech signals. Prior methods mainly focus on single-task speech restoration (SSR), such as speech denoising or speech declipping. However, SSR systems do not address the general speech restoration problem and show limited performance in some speech restoration tasks such as speech super-resolution. To overcome those limitations, we propose a general speech restoration (GSR) task that attempts to remove multiple distortions simultaneously. Furthermore, we propose VoiceFixer1, a generative framework to address the GSR task. VoiceFixer consists of an analysis stage and a synthesis stage to mimic the speech analysis and comprehension of the human auditory system. We employ a ResUNet to model the analysis stage and a neural vocoder to model the synthesis stage. We evaluate VoiceFixer with additive noise, room reverberation, low-resolution, and clipping distortions. Our baseline GSR model achieves a 0.499 higher mean opinion score (MOS) than the speech denoising SSR model. VoiceFixer further surpasses the GSR baseline model on the MOS score by 0.256. Moreover, we observe that VoiceFixer generalizes well to severely degraded real speech recordings, indicating its potential in restoring old movies and historical speeches.
references.authors: Fanhu Bie Dong Wang Jun Wang Thomas Fang Zheng
references.title: Detection and reconstruction of clipped speech for speaker recognition
references.venue: Speech Communication,
references.authors: Sawyer Birnbaum Volodymyr Kuleshov Zayd Enam Pang Wei Koh Stefano Ermon
references.title: Temporal film: Capturing long-range sequence dependencies with feature-wise modulations
references.authors: Albert S Bregman
references.title: Auditory scene analysis: The perceptual organization of sound
references.venue: MIT press,
references.authors: Benjamin Cauchi Ina Kodrasi Robert Rehr Stephan Gerlach Ante Jukic Timo Gerkmann Simon Doclo Stefan Goetze
references.title: Joint dereverberation and noise reduction using beamforming and a single-channel speech enhancement scheme
references.venue: In Proc. REVERB Challenge Workshop,
references.authors: Junyoung Chung Caglar Gulcehre KyungHyun Cho Yoshua Bengio
references.title: Empirical evaluation of gated recurrent neural networks on sequence modeling
references.venue: arXiv preprint arXiv:1412.3555,
references.authors: Djork-Arné Clevert Thomas Unterthiner Sepp Hochreiter
references.title: Fast and accurate deep network learning by exponential linear units
references.venue: arXiv preprint arXiv:1511.07289,
references.authors: Ross Cutler Ando Saabas Tanel Parnamaa Markus Loide Sten Sootla Marju Purin Hannes Gamper Sebastian Braun Karsten Sorensen Robert Aichner
references.title: Acoustic echo cancellation challenge
references.venue: In INTERSPEECH,
references.authors: Alexandre Defossez Gabriel Synnaeve Yossi Adi
references.title: Real time speech enhancement in the waveform domain
references.venue: arXiv preprint arXiv:2006.12847,
references.authors: Adoram Erell Mitch Weintraub
references.title: Estimation using log-spectral-distance criterion for noise-robust speech recognition
references.venue: In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing,
references.authors: Simon Godsill Peter Rayner Olivier Cappé
references.title: Digital audio restoration
references.venue: In Applications of Digital Signal Processing to Audio and Acoustics,
references.authors: Timothy D Griffiths Jason D Warren
references.title: The planum temporale as a computational hub
references.venue: Trends in Neurosciences,
references.authors: Archit Gupta Brendan Shillingford Yannis Assael Thomas C Walters
references.title: Speech bandwidth extension with wavenet
references.venue: In IEEE Workshop on Applications of Signal Processing to Audio and Acoustics,
references.authors: Kun Han Yuxuan Wang DeLiang Wang William S Woods Ivo Merks Tao Zhang
references.title: Learning spectral mapping for speech dereverberation and denoising
references.venue: IEEE/ACM Transactions on Audio, Speech, and Language Processing,
references.authors: Xiang Hao Xiangdong Su Shixue Wen Zhiyu Wang Yiqian Pan Feilong Bao Wei Chen
references.title: Masking and inpainting: A two-stage speech enhancement approach for low snr and nonstationary noise
references.venue: In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing,
references.authors: Yanxin Hu Yun Liu Shubo Lv Mengtao Xing Shimin Zhang Yihui Fu Jian Wu Bihong Zhang Lei Xie
references.title: DCCRN: Deep complex convolution recurrent network for phase-aware speech enhancement
references.authors: Kevin Jarrett Koray Kavukcuoglu Marc’Aurelio Ranzato Yann LeCun
references.title: What is the best multistage architecture for object recognition
references.venue: In International Conference on Computer Vision, pp
references.authors: Lauri Juvela Bajibabu Bollepalli Junichi Yamagishi Paavo Alku
references.title: GELP: GAN-Excited linear prediction for speech synthesis from mel-spectrogram
references.authors: Thomas Kailath
references.title: Lectures on Wiener and Kalman filtering
references.venue: In Lectures on Wiener and Kalman Filtering,
references.authors: Hamidreza Baradaran Kashani Ata Jodeiri Mohammad Mohsen Goodarzi Shabnam Gholamdokht Firooz
references.title: Image to image translation based on convolutional neural network approach for speech declipping
references.authors: Dan Kennedy-Higgins
references.title: Neural and cognitive mechanisms affecting perceptual adaptation to distorted speech
references.venue: PhD thesis,
references.authors: Keisuke Kinoshita Marc Delcroix Takuya Yoshioka Tomohiro Nakatani Emanuel Habets Reinhold Haeb-Umbach Volker Leutnant Armin Sehr Walter Kellermann Roland Maas
references.title: The REVERB challenge: A common evaluation framework for dereverberation and recognition of reverberant speech
references.venue: In IEEE Workshop on Applications of Signal Processing to Audio and Acoustics,
references.authors: Sran Kitić Nancy Bertin Rémi Gribonval
references.title: Sparsity and cosparsity for audio declipping: a flexible non-convex approach
references.venue: In International Conference on Latent Variable Analysis and Signal Separation,
references.authors: Jungil Kong Jaehyeon Kim Jaekyoung Bae
references.title: HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis
references.venue: arXiv preprint arXiv:2010.05646,
references.authors: Qiuqiang Kong Yong Xu Iwona Sobieraj Wenwu Wang Mark D Plumbley
references.title: Sound event detection and time–frequency segmentation from weakly labelled data
references.venue: IEEE/ACM Transactions on Audio, Speech, and Language Processing,
references.authors: Qiuqiang Kong Yin Cao Haohe Liu Keunwoo Choi Yuxuan Wang
references.title: Decoupling magnitude and phase estimation with deep resunet for music source separation
references.venue: In The International Society for Music Information Retrieval,
references.authors: Qiuqiang Kong Haohe Liu Xingjian Du Li Chen Rui Xia Yuxuan Wang
references.title: Speech enhancement with weakly labelled data from audioset
references.venue: arXiv preprint arXiv:2102.09971,
references.authors: Juho Kontio Laura Laaksonen Paavo Alku
references.title: Neural network-based artificial bandwidth expansion of speech
references.venue: IEEE/ACM Transactions on Audio, Speech, and Language Processing,
references.authors: Volodymyr Kuleshov S Zayd Enam Stefano Ermon
references.title: Audio super resolution using neural networks
references.venue: arXiv preprint arXiv:1708.00853,
references.authors: Kundan Kumar Rithesh Kumar Thibault de Boissiere Lucas Gestin Wei Zhen Teoh Jose Sotelo Alexandre de Brébisson Yoshua Bengio Aaron Courville
references.title: Melgan: Generative adversarial networks for conditional waveform synthesis
references.venue: arXiv preprint arXiv:1910.06711,
references.authors: Rithesh Kumar Kundan Kumar Vicki Anand Yoshua Bengio Aaron Courville
references.title: NU-GAN: High resolution neural upsampling with gan
references.venue: arXiv preprint arXiv:2010.11362,
references.authors: Frances Y Kuo Ian H Sloan
references.title: Lifting the curse of dimensionality
references.venue: Notices of the American Mathematical Society,
references.authors: Jonathan Le Roux Scott Wisdom Hakan Erdogan John R Hershey
references.title: SDR–half-baked or well done
references.venue: In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing,
references.authors: Junhyeok Lee Seungu Han
references.title: Nu-wave: A diffusion probabilistic model for neural audio upsampling
references.venue: arXiv preprint arXiv:2104.02321,
references.authors: Kehuang Li Chin-Hui Lee
references.title: A deep neural network approach to speech bandwidth expansion
references.venue: In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing,
references.authors: Yunpeng Li Marco Tagliasacchi Oleg Rybakov Victor Ungureanu Dominik Roblek
references.title: Real-time speech frequency bandwidth extension
references.venue: In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing,
references.authors: Teck Yian Lim Raymond A Yeh Yijia Xu Minh N Do Mark Hasegawa-Johnson
references.title: Timefrequency networks for audio super-resolution
references.venue: In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing,
references.authors: Ju Lin Yun Wang Kaustubh Kalgaonkar Gil Keren Didi Zhang Christian Fuegen
references.title: A two-stage approach to speech bandwidth extension
references.authors: Haohe Liu Lei Xie Jian Wu Geng Yang
references.title: Channel-wise subband input for better voice and accompaniment separation on high resolution music
references.venue: arXiv preprint arXiv:2008.05216,
references.authors: Philipos C Loizou
references.title: Speech enhancement: theory and practice
references.venue: CRC press,
references.authors: Yi Luo Nima Mesgarani
references.title: Conv-tasnet: Surpassing ideal time–frequency magnitude masking for speech separation
references.venue: IEEE/ACM Transactions on Acoustics, Speech, and Signal Processing,
references.authors: Craig Macartney Tillman Weyde
references.title: Improved speech enhancement with the Wave-U-Net
references.venue: arXiv preprint arXiv:1811.11307,
references.authors: Wolfgang Mack Emanuël AP Habets
references.title: Declipping speech using deep filtering
references.venue: In IEEE Workshop on Applications of Signal Processing to Audio and Acoustics,
references.authors: Rainer Martin
references.title: Spectral subtraction based on minimum statistics
references.authors: Annamaria Mesaros Toni Heittola Tuomas Virtanen
references.title: A multi-device dataset for urban acoustic scene classification
references.venue: arXiv preprint arXiv:1807.09840,
references.authors: Masanori Morise Fumiya Yokomori Kenji Ozawa
references.title: World: a vocoder-based high-quality speech synthesis system for real-time applications
references.venue: IEICE Transactions on Information and Systems,
references.authors: Anna K Nábělek Tomasz R Letowski Frances M Tucker
references.title: Reverberant overlap-and self-masking in consonant identification
references.venue: The Journal of the Acoustical Society of America,
references.authors: Yoshihisa Nakatoh Mineo Tsushima Takeshi Norimatsu
references.title: Generation of broadband speech from narrowband speech based on linear mapping
references.venue: Electronics and Communications in Japan,
references.authors: Arun Narayanan DeLiang Wang
references.title: Ideal ratio mask estimation using deep neural networks for robust speech recognition
references.venue: In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing,
references.authors: Patrick A Naylor Nikolay D Gaubitch
references.title: Speech dereverberation
references.venue: Springer Science & Business Media,
references.authors: Aaron van den Oord Sander Dieleman Heiga Zen Karen Simonyan Oriol Vinyals Alex Graves Nal Kalchbrenner Andrew Senior Koray Kavukcuoglu
references.title: Wavenet: A generative model for raw audio
references.venue: arXiv preprint arXiv:1609.03499,
references.authors: Santiago Pascual Antonio Bonafonte Joan Serra
references.title: SEGAN: Speech enhancement generative adversarial network
references.venue: arXiv preprint arXiv:1703.09452,
references.authors: Wei Ping Kainan Peng Kexin Zhao Zhao Song
references.title: Waveflow: A compact flow-based model for raw audio
references.venue: In International Conference on Machine Learning,
references.authors: Adam Polyak Lior Wolf Yossi Adi Ori Kabeli Yaniv Taigman
references.title: High fidelity speech regeneration with application to speech enhancement
references.venue: In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing,
references.authors: Ryan Prenger Rafael Valle Bryan Catanzaro
references.title: Waveglow: A flow-based generative network for speech synthesis
references.venue: In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing,
references.authors: Yi Ren Yangjun Ruan Xu Tan Tao Qin Sheng Zhao Zhou Zhao Tie-Yan Liu
references.title: FastSpeech: Fast, robust and controllable text to speech
references.authors: Lucas Rencker Francis Bach Wenwu Wang Mark D Plumbley
references.title: Sparse recovery and dictionary learning from nonlinear compressive measurements
references.venue: IEEE Transactions on Signal Processing,
references.authors: Dayana Ribas Emmanuel Vincent José Ramón Calvo
references.title: A study of speech distortion conditions in real scenarios for speech processing applications
references.venue: In IEEE Spoken Language Technology Workshop,
references.authors: Antony W Rix John G Beerends Michael P Hollier Andries P Hekstra
references.title: Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs
references.venue: In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing,
references.authors: Olaf Ronneberger Philipp Fischer Thomas Brox
references.title: U-net: Convolutional networks for biomedical image segmentation
references.venue: In International Conference on Medical image computing and computerassisted intervention,
references.authors: Pascal Scalart
references.title: Speech enhancement based on a priori signal to noise estimation
references.venue: In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing Conference Proceedings,
references.authors: Boaz Schwartz Sharon Gannot Emanuël AP Habets
references.title: Online speech dereverberation using kalman filter and em algorithm
references.venue: IEEE/ACM Transactions on Audio, Speech, and Language Processing,
references.authors: Jonathan Shen Ruoming Pang Ron J Weiss Mike Schuster Navdeep Jaitly Zongheng Yang Zhifeng Chen Yu Zhang Yuxuan Wang Rj Skerrv-Ryan
references.title: Natural TTS synthesis by conditioning wavenet on Mel spectrogram predictions
references.venue: In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing,
references.authors: Yao Shi Hui Bu Xin Xu Shaoji Zhang Ming Li
references.title: Aishell-3: A multi-speaker mandarin tts corpus and the baselines
references.venue: arXiv preprint arXiv:2010.11567,
references.authors: Xiaofeng Shu Yehang Zhu Yanjie Chen Li Chen Haohe Liu Chuanzeng Huang Yuxuan Wang
references.title: Joint echo cancellation and noise suppression based on cascaded magnitude and complex mask estimation
references.venue: arXiv preprint arXiv:2107.09298,
references.authors: Serkan Sulun Matthew EP Davies
references.title: On filter generalization for music bandwidth extension using deep neural networks
references.venue: IEEE Journal of Selected Topics in Signal Processing,
references.authors: Ke Tan Yong Xu Shi-Xiong Zhang Meng Yu Dong Yu
references.title: Audio-visual speech separation and dereverberation with a two-stage multimodal network
references.venue: IEEE Journal of Selected Topics in Signal Processing,
references.authors: Qiao Tian Yi Chen Zewang Zhang Heng Lu Linghui Chen Lei Xie Shan Liu
references.title: TFGAN: Time and frequency domain based generative adversarial network for high-fidelity speech synthesis
references.authors: Cassia Valentini-Botinhao
references.title: Noisy speech database for training speech enhancement algorithms and TTS models
references.authors: Tim Van den Bogaert Simon Doclo Jan Wouters Marc Moonen
references.title: Speech enhancement with multichannel wiener filter techniques in multimicrophone binaural hearing aids
references.venue: The Journal of the Acoustical Society of America,
references.authors: Charles Van Winkle
references.title: Audio analysis and spectral restoration workflows using adobe audition
references.venue: In Audio Engineering Society Conference: 33rd International Conference: Audio Forensics-Theory and Practice,
references.authors: Emmanuel Vincent Shinji Watanabe Aditya Arie Nugraha Jon Barker Ricard Marxer
references.title: An analysis of environment, microphone and data simulation mismatches in robust speech recognition
references.venue: Computer Speech & Language,
references.authors: Heming Wang DeLiang Wang
references.title: Towards robust speech super-resolution
references.venue: IEEE/ACM Transactions on Audio, Speech, and Language Processing,
references.authors: Hong Wang Fumitada Itakura
references.title: Dereverberation of speech signals based on sub-band envelope estimation
references.venue: IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences,
references.authors: Wenfu Wang Shuang Xu Bo Xu
references.title: First step towards end-to-end parametric tts synthesis: Generating spectral parameters with neural attention
references.venue: In INTERSPEECH,
references.authors: Zhou Wang Alan C Bovik Hamid R Sheikh Eero P Simoncelli
references.title: Image quality assessment: from error visibility to structural similarity
references.venue: IEEE Transactions on Image Processing,
references.authors: Donald S Williamson DeLiang Wang
references.title: Time-frequency masking in the complex domain for speech dereverberation and denoising
references.venue: IEEE/ACM Transactions on Audio, Speech, and Language Processing,
references.authors: Junichi Yamagishi Christophe Veaux Kirsten MacDonald
references.title: Cstr vctk corpus: English multispeaker corpus for cstr voice cloning toolkit
references.authors: Ryuichi Yamamoto Eunwoo Song Jae-Min Kim
references.title: Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram
references.venue: In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing,
references.authors: Chengzhu Yu Heng Lu Na Hu Meng Yu Chao Weng Kun Xu Peng Liu Deyi Tuo Shiyin Kang Guangzhi Lei
references.title: Durian: Duration informed attention network for multimodal synthesis
references.venue: arXiv preprint arXiv:1909.01700,
references.authors: Pavel Záviška Pavel Rajmic Ondřej Mokrỳ Zdeněk
references.title: Prŭša. A proper version of synthesis-based sparse audio declipper
references.venue: In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing,
references.authors: Pavel Záviška Pavel Rajmic Jiřı́ Schimmel
references.title: Psychoacoustically motivated audio declipping based on weighted l 1 minimization
references.venue: 42nd International Conference on Telecommunications and Signal Processing,
references.authors: Pavel Záviška Pavel Rajmic Alexey Ozerov Lucas Rencker
references.title: A survey and an extensive evaluation of popular audio declipping methods
references.venue: IEEE Journal of Selected Topics in Signal Processing,
references.authors: Yan Zhao Zhong-Qiu Wang DeLiang Wang
references.title: Two-stage deep learning for noisy-reverberant speech enhancement
references.venue: IEEE/ACM Transactions on Audio, Speech, and Language Processing,
references.authors: Kashani
references.title: 2019) treat the declipping as an image-toimage translation problem and utilize the UNet to perform spectral mapping. Currently, most of the state-of-the-art methods are unsupervised (Záviška et al., 2020
references.venue: neural network (DNN) (Bie et al.,
references.authors: Záviška
references.title: 2020) believes supervised models still have the potential for better declipping performance. Speech denoising Conventional methods are efficient and effective on dealing with stationary noise, such as spectral substraction (Martin, 1994) and Wiener and Kalman filtering (Kailath, 1981) By comparison, deep learning based models such as Conv-TasNet
references.authors: Han
references.title: 2015) used a fully connected DNN to learn a spectral mapping from reverberant speech to clean speech. In Williamson & Wang (2017), similar to the masking-based denoising methods, they proposed to perform dereverberation using a time-frequency mask
references.authors: Wang
references.title: The low-resolution and target data pairs are obtained by transforming 44.1 kHz sample rate utterances in target speech data VCTK-Test to a lower sample rate u. To achieve that, we first convolve the speech data with an order 8 Chebyshev type I lowpass filter with the
sections.heading: 1 INTRODUCTION
sections.text: Speech restoration is a process to restore degraded speech signals to high-quality speech signals. Speech restoration is an important research topic since distortions are ubiquitous. For example, speech is usually surrounded by background noise, blurred by room reverberations, or recorded by low-quality devices (Godsill et al., 2002). Those distortions degrade the perceptual quality of speech for human listeners. Speech restoration has a wide range of applications such as online meeting (Defossez et al., 2020), hearing aids (Van den Bogaert et al., 2009), and audio editting (Van Winkle, 2008). Still, speech restoration remains a challenging problem due to the large variety of distortions in the world.
  Previous works in speech restoration mainly focus on single task speech restoration (SSR), which deals with only one type of distortion at a time. For example, speech denoising (Loizou, 2007), speech dereveberation (Naylor & Gaubitch, 2010), speech super-resolution (Kuleshov et al., 2017), or speech declipping (Záviška et al., 2020). However, in the real world, speech signal can be degraded by several different distortions simultaneously, which means previous SSR systems oversimplify the speech distortion types (Kashani et al., 2019; Lin et al., 2021; Kuleshov et al., 2017; Birnbaum et al., 2019). The mismatch between the training data used in SSR and the testing data from the real world degrades the speech restoration performance.
  To address the mismatch problem, we propose a new task called general speech restoration (GSR), which aims at restoring multiple distortions in a single model. A numerous studies (Cutler et al., 2021; Cauchi et al., 2014; Han et al., 2015) have reported the benefits of jointly training multiple speech restoration tasks. Nevertheless, performing GSR using one-stage systems still suffer from the problems in each SSR task. For example, on generative tasks such as speech superresolution (Kuleshov et al., 2017), one-stage models tend to overfitting the filter (Sulun & Davies, 2020) used during training. Based on these observations, we propose a two-stage system called VoiceFixer.
  1Restoration samples: https://anonymous20211004.github.io/demo-vf/
  The design of VoiceFixer is motivated by the biological mechanisms of human hearing when restoring distorted speech. Intuitively, if a person tries to identify a strongly distorted voice, his/her brain can do the recovery by utilizing both the distorted speech signal and the prior knowledge of the language. As shown in Figure 1, the speech distortion perception is modeled by neuroscientists as a two-stage process, including an auditory scene analysis stage (Bregman, 1994), and a high level comprehension/synthesis stage (Griffiths & Warren, 2002). In the analysis stage, the sound information is first transformed into acoustic features by primary auditory cortex (PAC). Then planum temporale (PT), the cortical area posterior to the auditory cortex, acts as a computational hub by segregating and matching the acoustic features to low level spectrotemporal representations. In the synthesis stage, a high order cortical area is hypothesised to perform the high level perception tasks (Griffiths & Warren, 2002; Kennedy-Higgins, 2019). Our proposed VoiceFixer systems model the analysis stage with spectral transformations and a deep residual UNet, and the synthesis stage with a convolutional vocoder trained using adversarial losses. One advantage of the two-stage VoiceFixer is that the analysis and synthesis stages can be trained separately. Two-stage methods have also been successfully applied to the speech synthesis task (Wang et al., 2016; Ren et al., 2019) where acoustic models and vocoders are trained separately.
  VoiceFixer is the first GSR model that is able to restore a wide range of low-resolution speech sampled from 2 kHz to 44.1 kHz, which is different from previous studies working on constant sampling rates (Lim et al., 2018; Wang & Wang, 2021; Lee & Han, 2021). To the best of our knowledge, VoiceFixer is the first model that jointly performs speech denoising, speech dereverberation, speech super-resolution, and speech declipping in a unified model.
  The rest of this paper is organized as follows. Section 2 introduces the formulations of speech distortions. Section 3 describes the design of VoiceFixer. Section 4 discusses the evaluation results. Appendixes introduce related works and show speech restoration demos.
sections.heading: 2 PROBLEM FORMULATIONS
sections.text: We denote a segment of a speech signal as s ∈ RL, where L is the samples number in the segment. We model the distortion process of the speech signal as a function d(·). The degraded speech x ∈ RL can be written as:
  x = d(s). (1)
  Speech restoration is a task to restore high-quality speech ŝ from x:
  ŝ = f(x) (2)
  where f(·) is the restoration function and can be viewed as a reverse process of d(·). The target is to estimate s by restoring ŝ from the observed speech x. Recently, several deep learning based one-stage methods have been proposed to model f(·) such as fully connected neural networks, recurrent neural networks, and convolutional neural networks. Detailed introductions can be found in Appendix A.
  Distortion modeling is an important step to simulate distorted speech when building speech restoration systems. Several previous works model distortions in a sequential order (Vincent et al., 2017; Tan et al., 2020; Zhao et al., 2019). Similarly, we model the distortion d(·) as a composite function:
  d(x) = d1 ◦ d2 ◦ ...dQ(x), dq ∈ D, q = 1, 2, ..., Q, (3)
  where ◦ stands for function composition and Q is the number of distortions to consist d(·). Set D = {dv(·)}Vv=1 is the set of distortion types where V is the total number of types. Equation 3 describes the procedure of compounding different distortions from D in a sequential order. We introduce four speech distortions as follows.
  Additive noise is one of the most common distortion and can be modeled by the addition between speech s and noise n ∈ RL:
  dnoise(s) = s + n. (4)
  Reverberation is caused by the reflections of signal in a room. Reverberation makes speech signals sound distant and blurred. It can be modeled by convolving speech signals with a room impulse response filter (RIR) r:
  drev(s) = s ∗ r, (5)
  where ∗ stands for convolution operation. Low-resolution distortions refer to audio recordings that are recorded in low sampling rates or with limited bandwidth. There are many causes for low-resolution distortions. For example, when microphones have low responses in high-frequency, or audio recordings are compressed to low sampling rates, the high frequencies information will be lost. We follow the description in Wang & Wang (2021) to produce low-resolution distortions but add more filter types (Sulun & Davies, 2020). After designing a low pass filter h, we first convolve it with s to avoid the aliasing phenomenon. Then we perform resampling on the filtered result from the original sampling rate o to a lower sampling rate u:
  dlow res(s) = Resample(s ∗ h, o, u), (6)
  Clipping distortions refer to the clipped amplitude of audio signals, which are usually caused by low-quality microphones. Clipping can be modeled by restricting signal amplitudes within [−η,+η]:
  dclip(s) = max(min(s, η),−η), η ∈ [0, 1]. (7)
  In the frequency domain, the clipping effect produces harmonic components in the high-frequency part and degrades speech intelligibility accordingly.
sections.heading: 3 METHODOLOGY
sections.heading: 3.1 ONE-STAGE SPEECH RESTORATION MODELS
sections.text: Previous deep learning based speech restoration models are usually in one stage. That is, a model predicts restored speech ŝ from input x directly:
  f : x→ ŝ. (8)
  The mapping function f(·) can be modeled by time domain speech restoration systems such as onedimensional convolutional neural networks (Luo & Mesgarani, 2019) or frequency domain systems such as mask-based (Narayanan & Wang, 2013) methods:
  Ŝ = (Fsp(|X| ; θ) (|X|+ ))ej∠X . (9)
  where X is the short-time fourier transform (STFT) of x and is a small positive constant. X has a shape of T×F where T is the number of frame and F is the number of frequency bins. The output of the mask estimation function F (·; θ) is multiplied by the magnitude spectrogram |X| to produce the target spectrogram estimation Ŝ. Then, inverse short-time fourier transform (iSTFT) is applied on Ŝ to obtain ŝ. The one-stage speech restoration models are typically optimized by minimizing the mean absolute error (MAE) loss between the estimated spectrogram Ŝ and the target spectrogram S:
  L = ∥∥∥|Ŝ| − |S|∥∥∥
  1 (10)
  Previous one-stage models usually build on high-dimensional features such as time samples and the STFT spectrograms. However, Kuo & Sloan (2005) point out that the high-dimensional features will lead to exponential growth in search space. The model can work on the high-dimensional features under the premise of enlarging the model capacity but may also fail in challenging tasks. Therefore, it would be beneficial if we could build a system on more delicate low-dimensional features.
sections.heading: 3.2 VOICEFIXER
sections.text: In this study, we propose VoiceFixer, a two-stage speech restoration framework. Multi-stage methods have achieved state-of-the-art performance in many speech processing tasks (Jarrett et al., 2009; Tan et al., 2020). In speech restoration, our proposed VoiceFixer breaks the conventional one-stage system into a two-stage system:
  f : x 7→ z, (11)
  g : z 7→ ŝ. (12) Equation 11 denotes the analysis stage of VoiceFixer where a distorted speech x is mapped into a representation z. Equation 12 denotes the synthesis stage of VoiceFixer, which synthesize z to the restored speech ŝ. Through the two-stage processing, VoiceFixer mimics the human perception of speech described in Section 1.
sections.heading: 3.2.1 ANALYSIS STAGE
sections.text: The goal of the analysis stage is to predict the intermediate representation z, which can be used later to recover the speech signal. In our study, we choose mel spectrogram as the intermediate representation. Mel spectrogram has been widely used in many speech proceessing tasks (Shen et al., 2018; Kong et al., 2019; Narayanan & Wang, 2013). The frequency dimension of mel spectrogram is usually much smaller than that of STFT thus can be regarded as a way of feature dimension
  reduction. So, the objective of the analysis stage becomes to restore mel spectrograms of the target signals. The mel spectrogram restoration process can be written as the following equation,
  Ŝmel = fmel(Xmel;α) (Xmel + ), (13)
  where Xmel is the mel spectrogram of x. It is calculated by Xmel = |X|W where W is a set of mel filter banks with shape of F × F ′. The columns of W are not divided by the width of their mel bands, i.e., area normalization, because this will make the restoration model difficult to recover the high-frequency part. The mapping function fmel(·;α) is the mel restoration mask estimation model parameterized by α. The output of fmel is multiplied by Xmel to predict the target mel spectrogram.
  We use ResUNet (Kong et al., 2021a) to model the analysis stage as shown in Figure 3a, which is an improved UNet (Ronneberger et al., 2015). The ResUNet consists of several encoder and decoder blocks. There are skip connections between encoder and decoder blocks at the same level. As is shown in Figure 3b, both encoder and decoder block share the same structure, which is a series of residual convolutions (ResConv). Each convolutional layer in ResConv consists of a batch normalization (BN), a leakyReLU activation, and a linear convolutional operation. The encoder blocks apply average pooling for downsampling. The decoder blocks apply transpose convolution for upsampling. In addition to ResUNet, we implement the analysis stage with fully connected deep neural network (DNN), and bidirectional gated recurrent units (BiGRU) (Chung et al., 2014) for comparison. The DNN consists of six fully connected layers. The BiGRU has similar structures with DNN except for replacing the last two layers of DNN into bi-directional GRU layers.
  The details of these three models are discussed in Appendix B.1. We will refer to ResUNet as UNet later for abbreviation. We optimize the model in the analysis stage using the MAE loss between the estimated mel spectrogram Ŝmel and the target mel spectrogram Smel:
  Lana = ∥∥∥Ŝmel − Smel∥∥∥
  1 (14)
sections.heading: 3.2.2 SYNTHESIS STAGE
sections.text: The synthesis stage is realized by a neural vocoder that synthesizes the mel spectrogram into waveform as denoted in Equation 15:
  ŝ = g(Xmel;β), (15)
  where g(·;β) stands for the vocoder model parameterized by β. We employ a recently proposed nonautoregressive model, time and frequency domain based generative adversarial network (TFGAN), as the vocoder.
  Figure 4 shows the detailed architecture of TFGAN, in which the input mel spectrogram Xmel will first pass through a condition network CondNet, which contains N1 one-dimensional convolution layers with exponential linear unit activations (Clevert et al., 2015). Then, in UpNet, the feature is upsampledN2 times with ratios of s0,s1,..., and sN2−1 using UpsampleBlock and ResStacks. Within the UpsampleBlock, the input is first passed through a leakyReLU activation and then fed into a sinusoidal function, which output is added to its input to remove periodic artifacts in breathing part of speech. Then, the output is bifurcated into two branches for upsampling. One branch repeats the samples sn times followed by a one-dimensional convolution. The other branch uses a stride sn transpose convolution. The output of the repeat and transpose convolution branches are added together as the output of UpsampleBlock. ResStacks module contains two dilated convolution layers with leakyReLU activations. The exponentially growing dilation in ResStack enable the model to capture long range dependencies. The TFGAN in our synthesis model applies N2 = 4. After four UpsampleBlock blocks with ratios [7, 7, 3, 3], each frame of the mel spectrogram is transformed into a sequence with 441 samples corresponding to 10 ms of audio sampled at 44.1 kHz.
  The training criteria of the vocoder consist of frequency domain loss LF , time domain loss LT , and weighted discriminator loss LD:
  Lsyn = LF + LT + λDLD, (16)
  The frequency domain loss LF is the combination of a mel loss Lmel and multi-resolution spectrogram losses:
  LF (ŝ, s) = λmelLmel(ŝ, s) + KF∑ k=1 (λscL (k) sc (ŝ, s) + λmagL (k) mag(ŝ, s)) (17)
  where Lsc and Lmag are the spectrogram losses calculated in the linear and log scale, respectively. There are KF different window sizes ranging from 64 to 4096 to calculate Lsc and Lmag so that the trained vocoder is tolerant over phase mismatch (Yamamoto et al., 2020; Juvela et al., 2019). Table 2 in Appendix B.2 shows the detailed configurations.
  Time domain loss is complementary to frequency domain loss to address problems such as periodic artifacts. Time domain loss combines segment loss L(k)seg , energy loss L (k) energy and phase loss L (k) phase:
  LT (ŝ, s) = KT∑ k=1 (λsegL (k) seg (ŝ, s)) + λenergyL (k) energy(ŝ, s) + λphaseL (k) phase(ŝ, s) (18)
  where segment loss L(k)seg , energy loss L (k) energy and phase loss L (k) phase are described in Equation 24, 25, and 26 of Appendix B.2. There are KT different window sizes ranging from 1 to 960 to calculate time domain loss at different resolutions. The details of window sizes are shown in Table 3 of Appendix B.2. The energy loss and phase loss have the advantage of alleviating metallic sounds.
  Discriminative training is an effective way to train neural vocoders (Kong et al., 2020; Kumar et al., 2019). In our study, we utilize a group of discriminators, including a multi-resolution time discriminator DT , a subband discriminator Dsub, and frequency discriminator DF :
  D(s) = RT∑ r=1 D (r) T (s) +Dsub(s) +DF (s) (19)
  LD(s, ŝ) = min g max D
  (Es(log(D(s))) + Eŝ(log(1−D(ŝ)))). (20)
  The multi-resolution discriminators DT take signals from RT kinds of time resolutions after average pooling as input. The subband discriminator Dsub performs subband decomposition (Liu et al., 2020) on the waveform, producing four subband signals to feed into four T-discrminators, respectively. Frequency discriminator DF takes the linear spectrogram as input and outputs real or fake labels. The bottom part of Figure 4 shows the main idea of T-discriminator and F-discriminator. Appendix B.2 describes the detailed discriminator architectures.
  There are two advantages of using neural vocoder in the synthesis stage. First, neural vocoder trained using a large amount of speech data contains prior knowledge on the structural distribution of speech signals, which is crucial to the restoration of distorted speech. The amount of training data of vocoder is more than that used in conventional SSR methods with limited speaker numbers. Second, the neural vocoder typically takes the mel spectrogram as input, resulting in fewer feature dimensions than the STFT features. The reduction in dimension helps to lower computational costs and achieve better performance in the analysis stage.
sections.heading: 4 EXPERIMENTS
sections.heading: 4.1 DATASETS AND EVALUATION METRICS
sections.text: Training sets: The training speech datasets we use including VCTK (Yamagishi et al., 2019), AISHELL-3 (Shi et al., 2020), and HQ-TTS. We call the noise datasets used for training as VDNoise. To simulate the reverberations, we employ a set of RIRs to create an RIR-44k dataset. We use VCTK, VD-Noise, and the training part of RIR-44k to train the analysis stage. AISHELL-3, VCTK, and HQ-TTS datasets are used to train the vocoder in the synthesis stage. The details of those datasets and the RIRs simulation configurations are discussed in Appendix C.1.
  Test sets: We employ VCTK-Demand (Valentini-Botinhao et al., 2017) as the denoising test set and name it as DENOISE. We call our speech super-resolution, declipping, and dereverberation evaluation test sets as SR, DECLI, and DEREV, respectively. In addition, we create an ALL-GSR test set containing all distortions. We introduce the details of how we build these test sets in Appendix C.3.
  Evaluation metrics: The metrics we adopt include log-spectral distance (LSD) (Erell & Weintraub, 1990), wide band perceptual evaluation of speech quality (PESQ-wb) (Rix et al., 2001), structural similarity (SSIM) (Wang et al., 2004), and scale-invariant signal to noise ratio (SiSNR) (Le Roux et al., 2019). We use mean opinion scores (MOS) to subjevtively evaluate different systems.
  The output of the vocoder is not strictly aligned on sample level with the target, as is often the case in generative model (Kumar et al., 2020). This effect will degrade the metrics, especially for those calculated on time samples such as the SiSNR. So, to compensate the SiSNR, we design similar metric, scale-invariant spectrogram to noise ratio (SiSPNR), to measure the discrepancies on the spectrograms. Details of the metrics are described in Appendix C.4.
sections.heading: 4.2 DISTORTIONS SIMULATION
sections.text: For the SSR task, we perform only one type of distortion for evaluation. For the GSR task, we first assume that D = {dnoise, drev, dlow res, dclip} because those distortions are the most common distortions in daily environment (Ribas et al., 2016). Second, we assume that Q ≤ 4 in Equation 3. In other words, each distortion in D appears at most one time. Then, we generate the distortions following a specific order drev, dclip, dlow res, and dnoise. These distortions are added randomly using random configurations.
sections.heading: 4.3 BASELINE SYSTEMS
sections.text: Table 5 in Appendix D summarizes all the experiments in this study. We implement several SSR and GSR systems using one-stage restoration models. For the GSR, we train a ResUNet model called GSR-UNet with all distortions. For the SSR models, we implement a Denoise-UNet for additive noise distortion, a Dereverb-UNet for reverberation distortion, a SR-UNet for low-resolution distortion, and a Declip-UNet for clipping distortion. For the SR task, we also include two stateof-the-art models, NuWave (Lee & Han, 2021) and SEANet (Li et al., 2021) for comparison. For declipping task, we compare with a state-of-the-art synthesis-based method SSPADE (Záviška et al.,
  2019a). To explore the impact of model size of the mel restoration model, we setup ResUNets with two sizes, UNet-S and UNet, which have one and four ResConv blocks in each encoder and decoder block, respectively.
sections.heading: 4.4 EVALUATION RESULTS
sections.text: Neural vocoder To evaluate the performance of the neural vocoder, we compare two baselines. The Target system denotes using the perfect s for evaluation. The Unprocessed system denotes using distorted speech x for evaluation. The Oracle-Mel system denotes using the mel spectrogram of s as input to the vocoder, which marks the performance of the vocoder. As shown in Table 1, the OracleMel system achieves a MOS score of 3.74, which is close to the Target MOS of 3.95, indicating that the vocoder performs well in the synthesis task.
  General speech restorations: Table 1 shows the evaluation results on ALL-GSR test set. Figure 5 shows the box plot of the MOS scores of these systems. The GSR-UNet outperforms the two SSR models, Denoise-UNet and Dereverb-UNet by a large margin. It surpasses Denoise-UNet model by 0.5 on MOS score, which suggests the GSR model is more powerful than the SSR model on this test set. For convenience, we denote VoiceFixer as VF in tables and figures. We observe that the VF-UNet model achieves the highest MOS score and LSD score. Specifically, VF-UNet obtains 0.256 higher MOS score than that of GSR-UNet. This result indicates that VoiceFixer is better than ResUNet based one-stage model on overall quality. What’s more, we notice that the MOS score of VF-UNet is only 0.11 lower than the Oracle-Mel, demonstrating the good performance of the analysis stage. Among the VoiceFixer analysis models, the UNet front-end achieves the best. The VF-BiGRU model achieves similar subjective metrics with the VF-UNet model but has much lower MOS scores. This phenomenon shows that the improvement in subjective metrics in VoiceFixer is not always consistent with objective evaluation results.
  Super-resolution: Table 6 in Appendix D.1 shows the evaluation results on the super-resolution test set SR. For the 2 kHz, 4 kHz, and 8 kHz to 44.1 kHz super-resolution tasks, VF-UNet achieves a significantly higher LSD, SiSPNR and SSIM scores than other models. The LSD value of VF-UNet in 2 kHz sampling rate is still higher than the 8 kHz sampling rate score of GSR-UNet, SR-UNet, NuWave, and SEANet. This demonstrates the strong performance of VoiceFixer on dealing with low sampling rate cases. The VF-BiGRU model outperforms VF-UNet-S model on average scores for its better performance on low upsample-ratio cases. MOS box plot in Figure 6b shows that VF-UNet performs the best on 8 kHz to 44.1 kHz test set. Figure 6e shows the MOS score of Unprocessed is close to Target on 24 kHz to 44.1 kHz test set, meaning limited perceptual difference between the two sampling rates. On this test set, SEANet even achieves a higher MOS score than Target. That’s due to the high-frequency part it generate contains more energy than that of Target, making the results sound clearer.
  Denoising: We evaluate the speech denoising performance on the DENOISE test set and show results in Table 7 in Appendix D.1. We find that GSR-UNet preserves more details in the highfrequency part and has better PESQ and SiSPNR values than the denoising only SSR model DenoiseUNet. The reason might be that speech data augmentations and jointly performing super-resolution task can increase the generalization and inpainting ability of the model (Hao et al., 2020). The PESQ
  score of VF-UNet reaches 2.43, higher than SEGAN, WaveUNet, and the model trained with weakly labeled data in Kong et al. (2021b). The MOS evaluations in Figure 6f on speech denoising task also demonstrate that the result of VF-UNet sound comparable with one-stage speech denoising models.
  Declipping and dereveberation: Table 9 and Table 8 in Appendix D.1 show similar performance trends on the speech declipping and speech dereveberation. In both tasks, the SSR model DereverbUNet and Declip-UNet achieve the highest scores. The performance of GSR-UNet is slightly worse, but it is acceptable considering that GSR-UNet does not need extra training for each task. SSPADE performs better on SiSNR, but the PESQ and STOI scores are lower, especially in the 0.1 threshold case. The MOS score in Figure 6d shows that the clipping effect in the 0.25 threshold case is not easy to perceive, leading to high MOS scores across all methods. In Figure 6a, both Declip-UNet and VF-UNet achieve the highest objective scores on the 0.1 threshold clipping test set. On the dereverberation test set DEREV, VF-UNet achieves the highest MOS score 3.52.
sections.heading: 5 CONCLUSIONS
sections.text: In this work, we propose VoiceFixer, an effective approach for general speech restoration. The evaluation results show that VoiceFixer achieves leading performance across the general speech restoration, speech super-resolution, speech denoising, speech dereverberation, and speech declipping tasks. In the future, we will extend VoiceFixer to restore general audio signals, including music and general sounds.
sections.heading: REPRODUCIBILITY STATEMENT
sections.text: We make our code and datasets anonymously downloadable for painless reproducibility. Our pre-trained VoiceFixer and inference code are presented in https://github.com/ anonymous20211004/iclr2022-vf. The code for performing the experiments discussed in Section 4 is downloadable in https://github.com/anonymous20211004/ iclr2022-train-vf. The code can be used to conduct evaluations and generate reports on the metrics mentioned in Section 4.1 automatically. The NuWave is realized using the code open-sourced by Lee & Han (2021): https://github.com/mindslab-ai/nuwave. We reproduce SSPADE using the toolbox provided by Záviška et al. (2020) at https://rajmic.github.io/ declipping2020. Besides, we upload the training and testing sets mentioned in Section 4.1 to https://zenodo.org/record/5546723. The AISHELL-3 dataset is open-sourced at http://www.aishelltech.com/aishell_3. The HQ-TTS is a collection of datasets from openslr.org, as is described in Table 4 of Appendix C.1.
sections.heading: A APPENDIX A
sections.heading: A.1 SPEECH RESTORATION TASKS
sections.text: Speech super-resolution A lot of early studies (Nakatoh et al., 2002; Kontio et al., 2007) break super-resolution (SR) into spectral envelop estimation and excitation generation from the lowresolution part. At that time, the direct mapping from the low-resolution part to the high-resolution feature is not widely explored since the dimension of the high-resolution part is relatively high. Later, deep neural network (Li & Lee, 2015; Kuleshov et al., 2017) is introduced to perform SR. These approaches show better subjective quality comparing with traditional methods. To increase the modeling capacity, TFilm (Birnbaum et al., 2019) is proposed to model the affine transformation among each time block. Similarly, WaveNet also shows effectiveness in extending the bandwidth of a band-limited speech (Gupta et al., 2019). To utilize the information both from the time and frequency domain, Wang & Wang (2021) propose a time-frequency loss that can yield a balanced performance on different metrics. Recently, NU-GAN (Kumar et al., 2020) and NU-Wave (Lee & Han, 2021) pushed the target sample rate in SR to high fidelity, namely 48 kHz.
  Although employing deep neural networks in BWE shows promising results, the generalization capability of these methods is still limited. For example, previous approaches (Kuleshov et al., 2017) usually train and test models with a fixed setup, i.e., fixing the initial and target sampling rates. However, in real-world applications, speech bandwidth is not usually constant. In addition, since the real high-low quality speech pair is hard to collect, typically, we produces low-quality audio with lowpass or bandpass filters during training. In this case, systems tend to overfit specific filters. As discussed in Sulun & Davies (2020), when the kind of filter used during training and testing differ, the performance can fall considerably. To alleviate filter overfitting, Sulun & Davies (2020) propose to train models with multiple kinds of lowpass filters, by which the unseen filters can be handled properly.
  Speech declipping The methods for speech declipping can be categorized as supervised methods and unsupervised methods. The unsupervised, or blind methods usually perform declipping based on some generic regularization and assumption of what natural audio should look like, such as ASPADE (Kitić et al., 2015), dictionary learning (Rencker et al., 2019), and psychoacoustically motivated l1 minimization (Záviška et al., 2019b). The supervised models, mostly based on deep neural network (DNN) (Bie et al., 2015; Mack & Habets, 2019), are usually trained on clipped and unclipped data pairs. For example, Kashani et al. (2019) treat the declipping as an image-toimage translation problem and utilize the UNet to perform spectral mapping. Currently, most of the state-of-the-art methods are unsupervised (Záviška et al., 2020) because they are usually designed to work on different kinds of audio, while the supervised model mainly specialized on data similar to its training data. However, Záviška et al. (2020) believes supervised models still have the potential for better declipping performance.
  Speech denoising Conventional methods are efficient and effective on dealing with stationary noise, such as spectral substraction (Martin, 1994) and Wiener and Kalman filtering (Kailath, 1981) By comparison, deep learning based models such as Conv-TasNet (Luo & Mesgarani, 2019) show higher subjective score and robustness on complex cases. Recently, new schemes have emerged for training speech denoising models. SEGAN (Pascual et al., 2017) tried a generative way. Kong et al. (2021b) achieved a denoising model using only weakly labeled data. And Polyak et al. (2021) realized a denoising model using a regeneration approach.
  Speech dereveberation Some of the early methods in speech dereverberation, such as inverse filtering (Naylor & Gaubitch, 2010) and subband envelope estimation (Wang & Itakura, 1991), aiming at deconvolving the reverberant signal by estimating an inverse filter. However, the inverse filter is hard and not robust to estimate accurately. Other methods, like spectral substraction, is based on an overlap-masking (Nábělek et al., 1989) effect of reverberation. Schwartz et al. (2014) performed dereverberation using Kalman filter and expectation-maximization algorithm. Recently, deep learning based dereverberation methods have emerged as the state-of-the-art. Han et al. (2015) used a fully connected DNN to learn a spectral mapping from reverberant speech to clean speech. In Williamson & Wang (2017), similar to the masking-based denoising methods, they proposed to perform dereverberation using a time-frequency mask.
sections.heading: A.1.1 JOINT RESTORATION AND SYNTHETIC RESTORATION
sections.text: Joint restoration Many works have adopted the joint restoration approach to improving models. To make the acoustic echo cancellation (AEC) result sound cleaner, MC-TCN (Shu et al., 2021) proposed to jointly perform AEC and noise suppression at the same time. MC-TCN achieved a mean opinion score of 4.41, outperforming the baseline of AEC Challenge (Cutler et al., 2021) by 0.54. Moreover, in the REVERB challenge (Kinoshita et al., 2013), the test set has both reverberation and noise. Therefore, the proposed methods in the challenge should perform both denoising and dereverberation. In Han et al. (2015), the authors proposed to perform dereverberation and denoising within a single DNN and substantially outperform related methods regarding quality and intelligibility. However, previous joint processing usually involved only two sub tasks. In our study, we joint perform four or more tasks to achieve general restoration.
  Synthetic restoration Directly estimate the source signal from the observed mixture is hard especially when the SNR is low. Some studies adopted a regeneration approach. In Polyak et al. (2021), the authors utilized an ASR model, a pitch extraction model, and a loudness model to extract semantic level information from the speaker. Then these features were fed to an encode-decoder network to regenerate the speech signal. To maintain the consistency of speaker characteristics, it used an auxiliary identity network to compute the identity feature. Similar to synthetic speech restoration, TTS can be treated as the regeneration of speech from texts.
sections.heading: A.1.2 NEURAL VOCODER
sections.text: Vocoder, which maps the encoded speech features to the waveforms, is an indispensable component in speech synthesis. The most widely used input feature for vocoder is mel spectrogram. In recent years, since the emergence of WaveNet (Oord et al., 2016), neural network based vocoders demonstrate clear advantages over traditional parametric methods (Morise et al., 2016). Comparing with conventional methods, the quality of WaveNet is more closer to the human voice. Later, WaveRNN (Yu et al., 2019) is proposed to model the waveform with a single GRU. In this way, WaveRNN has much lower complexity comparing with WaveNet. However, the autoregressive nature of these models and deep structure make their inference process hard to be paralleled. To address this problem, non-autoregressive models like WaveGlow (Prenger et al., 2019) and WaveFlow (Ping et al., 2020) were proposed. Afterward, non-autoregressive GAN-based models such as MelGAN (Kumar et al., 2019) push the synthesis quality to a comparable level with autoregressive models. Recently, TFGAN (Tian et al., 2020) demonstrated strong capability in vocoding. Directed by multiple discriminators and loss functions, TFGAN was able to leverage information from both time domain and frequency domain. As a result, the synthesis quality of TFGAN is more natural and less metallic comparing with other GAN-based non-autoregressive models. In this work, we realize a universal vocoder based on TFGAN, which can reconstruct waveform from mel spectrogram of an arbitrary speaker with good perceptual quality. The pre-trained vocoder is available online to facilitate future studies and reproduce our work.
sections.heading: B APPENDIX B
sections.heading: B.1 DETAILS OF THE ANALYSIS STAGE
sections.text: The DNN and BiGRU we use are shown in Figure 7. DNN is a six layers fully connected network with BatchNorm and ReLU activations. The DNN accept each time step of the low-quality spectrogram as the input feature and output the restoration mask. Similarly, for the BiGRU model, we substitute some layers in DNN to a two-layer bidirectional GRU to capture the time dependency between time steps. To increase the modeling capacity of BiGRU, we expanded the input dimension of GRU to twice the mel frequency dimension with full connected networks.
  The detailed architecture of ResUNet is shown in Figure 3a. In the down-path, the input low-quality mel spectrogram will go through 6 encoder blocks, which includes a stack of L1 ResConv and a 2 × 2 average pooling. In ResConv, the outputs of ConvBlock and the residual convolution are added together as the output. ConvBlock is a typical two layers convolution with BatchNorm and leakyReLU activation functions. The kernel size of residual convolution and the convolution in ConvBlock is 1 × 1 and 3 × 3. Correspondingly, the decoder blocks have the symmetric structure of the encoder blocks. It first performs a transpose convolution with 2× 2 stride and 3× 3 kernels, which result is concatenated with the output of the encoder at the same level to form the input of the decoder. The decoder also contain L2 layers of ResConv. The output of the final decoder block is passed to a final ConvBlock to fit the output channel.
  We use Adam optimizer with β1 = 0.5, β2 = 0.999 and a 3e-4 learning rate to optimize the analysis stage of VoiceFixer. We treat the first 1000 steps as the warmup phase, during which the learning rate grows linearly from 0 to 3e-4. We decay the learning rate by 0.9 every 400 hours of training data. We perform an evaluation every 200 hours of training data. If we observe three consecutive evaluations with no improvement, we will interrupt the experiment.
  For all the STFT and iSTFT, we use the hanning window with a window length of 2048 and a hop length of 441. As all the audio we use is at the 44.1 kHz sample rate, the corresponding spectrogram size in this setting will be T ×1025, where T is the dimension of time frames. For mel spectrogram, the dimensions of the linear spectrogram are transformed into T × 128.
sections.heading: B.2 DETAILS OF THE SYNTHESIS STAGE
sections.text: As shown in Table 3, we use 7 kinds of STFT resolutions and 4 kinds of time resolution during the calculation of LF and LT . So KF = 7 in Equation 17 and KT = 4 in Equation 18.
  The mel loss Lmel, spectral convergence loss Lsc, STFT magnitude loss Lmag, segment loss Lseg, energy loss Lenergy, and phase loss Lphase are defined in Equation 21 26. The function v(·) is the
  k 1 2 3 4 5 6 7 win-length 4096 2048 1024 512 256 128 64 hop-length 2048 1024 512 256 128 64 32
  fft-size 8192 4096 2048 1024 512 256 128
  Table 2: STFT setup for different k in LF .
  k 1 2 3 4 frame-length 1 240 480 960 hop-length 1 120 240 480
  Table 3: Windowing setup for different k in LT .
  windowing function that divide time sample into w windows and compute mean value within each window, v(s)1×w = (mean(s0),mean(s1), ...,mean(sw−1)). Each sw stand for windowed s. ∆ stand for first difference.
  Lmel(ŝ, s) = ∥∥∥Ŝmel − Smel∥∥∥
  2 (21)
  Lsc(ŝ, s) = ∥∥∥|Ŝ| − |S|∥∥∥ F∥∥∥|Ŝ|∥∥∥
  F
  (22)
  Lmag(ŝ, s) = ∥∥∥log(|Ŝ|)− log(|S|)∥∥∥
  1 , (23)
  Lseg(ŝ, s) = ‖v(ŝw)− v(sw)‖1 , (24) Lenergy(ŝ, s) = ∥∥v(ŝw2)− v(s2w)∥∥1 , (25)
  Lphase(ŝ, s) = ∥∥∆v(ŝw2)−∆v(s2w)∥∥1 , (26)
  Table 9 and Table 8 show the structure of frequency and time domain discriminators. The subband discriminators Dsub and multi-resolution time discriminators D (r) T (s) use the structure of Tdiscriminator, which is a stack of one dimensional convolution with grouping and large kernal size. The frequency discriminator DF use the similar module ResConv similar to ResUNet shown in Figure 3b.
  F-discriminator Conv2d(1,32,kernal_size=(3,3))
  ResConv(32, 32, stride=1,kernal_size=(3,3)) ResConv(32, 32, stride=1,kernal_size=(3,3)) ResConv(32, 64, stride=2,kernal_size=(3,3)) ResConv(64, 64, stride=1,kernal_size=(3,3)) ResConv(64, 32, stride=2,kernal_size=(3,3)) ResConv(32, 32, stride=1,kernal_size=(3,3)) ResConv(32, 32, stride=2,kernal_size=(3,3)) ResConv(32, 32, stride=1,kernal_size=(3,3))
  Figure 9: The structure of F-discriminator.
  For the training of vocoder, we setting up the λD to λseg value in Equation 16, Equation 17, and Equation 18 as λD = 4.0, λmel = 50, λsc = 5.0, λmag = 5.0, λenergy = 100.0, λphase = 100.0, and λseg = 200.0
sections.heading: C APPENDIX C
sections.heading: C.1 DATASETS
sections.text: Clean speech CSTR VCTK corpus (Yamagishi et al., 2019) is a multi-speaker English corpus containing 110 speakers with different accents. We split it into a training part VCTK-Train and a testing part VCTK-Test. The version of VCTK we used is 0.92. To follow the data preparation strategy of Lee & Han (2021), only the mic1 microphone data is used for experiments, and p280 and p315 are omitted for the technical issues. For the remaining 108 speakers, the last 8 speakers, p360,p361,p362,p363,p364,p374,p376,s5 are splitted as test set VCTK-Test. Within the other 100 speakers, p232 and p257 are omitted because they are used later in the test set DENOISE, the remaining 98 speakers are defined as VCTK-Train. Except for the training of NuWave, all the utterances are resampled at the 44.1 kHz sample rate. AISHELL-3 is an open-source Hi-Fi mandarin speech corpus, containing 88035 utterances with a total duration of 85 hours. HQ-TTS dataset contains 191 hours of clean speech data collected from a serial of datasets on openslr.org. In Table 4, we include the details of HQ-TTS, including the URL and language types of each subset.
  Noise data One of the noise dataset we use come from VCTK-Demand (VD) (Valentini-Botinhao et al., 2017), a widely used corpus for speech denoising and noise-robust TTS training. This dataset contains a training part VD-Train and a testing part VD-Test, in which both contain two noisy set VD-Train-Noisy, VD-Test-Noisy and two clean speech set VD-Train-Clean, VD-Test-Clean. To obtain the noise data from this dataset, we minus each noisy data from VD-Train-Noisy with its corresponding clean part in VD-Train-Clean to get the final training noise dataset VD-Noise. The noise data are all resampled to 44.1 kHz. Another noise dataset we adopt is the TUT urban acoustic scenes 2018 dataset (Mesaros et al., 2018), which is originally used for the acoustic scene classification task of DCASE 2018 Challenge. The dataset contains 89 hours of high-quality recording from 10 acoustic scenes such as airport and shopping mall. The total amount of audio is divided into development DCASE-Dev and evaluation DCASE-Eval parts. Both of them contain audio from all cities and all acoustic scenes.
  Room impulse response We randomly simulated a collection of Room Impulse Response filters to simulate the 44.1 kHz speech room reverberation using an open-source tool 2. The meters of height, width, and length of the room are sampled randomly in a uniform distribution U(1, 12). The placement of the microphone is then randomly selected within the room space. For the placement of the sound source, we first determined the distance to the microphone, which is randomly sampled in a Gaussian distribution N(µ, σ2), µ = 2, σ = 4. If the sampled value is negative or greater than five meters, we will sample the distance again until it meets the requirement. After sampling the distance between the microphone and sound source, the placement of the sound source is randomly selected within the sphere centered at the microphone. The RT60 value we choose come from the uniform distribution U(0.05, 1.0). For the pickup pattern of the microphone, we randomly choose from omnidirectional and cardioid types. Finally, we simulated 43239 filters, in which we randomly split out 5000 filters as the test set RIR-Test and named other 38239 filters as RIR-Train.
  2https://github.com/sunits/rir_simulator_python
sections.heading: C.2 TRAINING DATA
sections.text: We describe the simulation of training data in Algorithm 1. S = {s(0), s(1), ..., s(i)}, N = {n(0),n(1), ...,n(i)}, and R = {r(0), r(1), ..., r(i)} are the speech dataset, noise dataset, and RIR dataset. We use several helper function to describe this algorithm. randomFilterType(·) is a function that randomly select a type of filter within butterworth, chebyshev, bessel, and ellipic. Resample(x, o1, u) is a resampling function that resample the one dimensional signal x from a original samplerate o1 to the target u samplerate. buildFilter(t, c, o2) is a filter design function that return a type t filter with cutoff frequency c and order o2. max(·),min(·), and abs(·) is the element wise maximum, minimum, and absolute value function. mean(·) calculate the mean value of the input.
  We first select a speech utterance s, a segment of noise n and a RIR filter r randomly from the dataset. Then with p1 probability, we add the reverberation effect using r. And with p2 probability, we add clipping effect with a clipping ratio η, which is sampled in a uniform distrubution U(ηlow, ηhigh). To produce low-resolution effect, after determining the filter type t, we randomly sample the cutoff frequency c and order o from the uniform distribution U(Clow, Chigh) and U(Olow, Ohigh). Then we perform convolution between x and the type t order o lowpass filter with cutoff frequency c. Finally the filtered data will be resampled twice, one is resample to c ∗ 2 samplerate and another is resample back to 44.1 kHz. We also perform the same lowpass filtering to the noise signal randomly. This operation is necessary because, if not, the model will overfit the pattern that the bandwidth of noise signal is always different from speech. In this case, the model will fail to remove noise when the bandwidth of noise and speech are similar. For the simulation of noisy environment, we randomly add the noise n into the speech signal x using a random SNR s ∼ U(Slow, Shigh). To fit the model with all energy levels, we randomly conduct a q ∼ U(Qlow, Qhigh) scaling to the input and target data pair. In our work, we choose the following parameters to perform this algorithm, p1 = 0.25, p2 = 0.25, p3 = 0.5, ηlow = 0.06, ηhigh = 0.9, Clow = 750, Chigh = 22050, Olow = 2, Ohigh = 10, Slow = −5, Shigh = 40, Qlow = 0.3, Qhigh = 1.0.
  Algorithm 1: Add random distortions to high-quality speech signal s In: s← S; n← N; r ← R Out: The high-quality speech s and its randomly distorted version x
  x = s; with p1 probability: x = x ∗ r ; /* Convolute with RIR filter */ with p2 probability:
  θ ∼ U(Θlow,Θhigh) ; /* Choose clipping ratio */ x = max(min(x, θ),−θ) ; /* Hard clipping */
  with p3 probability: t = randomFilterType() ; c ∼ U(Clow, Chigh) ; o ∼ U(Olow, Ohigh) ; /* Random cutoff and order */ x = x ∗ buildFilter(t, c, o) ; /* Low pass filtering */ x = Resample(Resample(x, 44100, c ∗ 2), c ∗ 2, 44100) ; /* Resample */ with p4 probability:
  n = n ∗ buildFilter(t, c, o) ; /* Low pass filtering on noise */ n = Resample(Resample(n, c ∗ 2), 44100) ; /* Resample */
  with p5 probability: s ∼ U(Slow, Shigh) ; q ∼ U(Qlow, Qhigh) ; /* Random SNR and scale */ n = nmean(abs(n))/mean(abs(x)) ; /* Normalize the energy of noise */ x = (x + n
  10s/20 ) ; /* Add noise */
  s = qs ; /* Scaling */ x = qx ; /* Scaling */
sections.heading: C.3 TESTING DATA
sections.text: Testing data is crucial for the evaluation for each kind of distortion. The testing data we use either come from open-sourced test set or simulated by ourselves.
  Super-resolution The simulation of the SR test set follows the work of (Kuleshov et al., 2017; Wang & Wang, 2021). The low-resolution and target data pairs are obtained by transforming 44.1 kHz sample rate utterances in target speech data VCTK-Test to a lower sample rate u. To achieve that, we first convolve the speech data with an order 8 Chebyshev type I lowpass filter with the u2 cutoff frequency. Then we subsample the signal to u sample rate using polyphase filtering. In this work, to test the performance on different sampling rate settings, u are set at 2 kHz, 4 kHz, 8 kHz, 16 kHz, and 24 kHz. We denote the corresponding five testing set as VCTK-4k, VCTK-4k, VCTK-8k, VCTK-16k, and VCTK-24k, respectively.
  Denoising For the denoising task, we adopt the open-sourced testing set DENOISE described in Appendix C.1. This test set contains 824 utterances from a female speaker and a male speaker. The type of noise data comprises a domestic noise, an office noise, noise in the transport scene, and two street noises. The test set is simulated at four SNR levels, which are 17.5 dB, 12.5 dB, 7.5 dB, and 2.5 dB. The original data is sampled at 48 kHz. We downsample it to 44.1 kHz to fit our experiments.
  Dereverberation The test set for dereverberation, DEREV, is simulated using VCTK-Test and RIRTest. For each utterance in VCTK-Test, we first randomly select an RIR from RIR-Test, then we calculate the convolution between the RIR and utterance to build the reverberant speech. Finally, we build 2937 reverberant and target data pairs.
  Declipping DECLI, the evaluation set for declipping, is also constructed based on VCTK-Test. We perform clipping on VCTK-Test following the equation in Section 2 and choose 0.25, 0.1 as the two setups for the clipping ratio. This result in two declipping test sets with different levels, each containing 2937 clipped speech and target audios.
  General speech restoration To evaluate the performance on GSR, we simulate a test set ALLGSR comprising of speech with all kinds of distortion. The clean speeches and noise data used to build ALL-GSR is VCTK-Test and DCASE-Eval. The simulation procedure of ALL-GSR is almost the same to the training data simulation described in Section 4.2. In total, 501 three seconds long utterances are produced in this test set.
  MOS Evaluation We select a small portion from the test sets to carry out MOS evaluation for each one. In SR, DECLI, and DEREV, we select 38 utterances out for human ratings. In DENOISE and ALL-GSR, we randomly choose 42 and 51 utterances.
sections.heading: C.4 EVALUATION METRICS
sections.text: Log-spectral distance LSD is a commonly used metrics on the evaluation of super-resolution performance (Kumar et al., 2020; Lee & Han, 2021; Wang & Wang, 2021). For target signal s and output estimate ŝ, LSD can be computed as Equation 27, where S and Ŝ stand for the magnitude spectrogram of s and ŝ.
  LSD(S, Ŝ) = 1
  T ΣTt=1
  √ 1
  F ΣFf=1 log10(
  S(f, t)2 Ŝ(f, t)2 )2 (27)
  Perceptual evaluation of speech quality PESQ is widely used in speech restoration literature as their evaluation metrics (Pascual et al., 2017; Hu et al., 2020). It was originally developed to model the subjective test commonly used in telecommunication. PESQ provides a score ranging from -0.5 to 4.5 and the higher the score, the better quality a speech has. In our work, we used an open-sourced implementation of PESQ to compute these metrics. Since PESQ only works on a 16 kHz sampling rate, we performed a 16 kHz downsampling to the output 44.1k audio before evaluation.
  Structural similarity SSIM (Wang et al., 2004) is a metrics in image super-resolution. It addresses the shortcoming of pixel-level metrics by taking the image texture into account. We match the implementation of SSIM in (Wang et al., 2004) with ours and compute SSIM as Equation 28, where µS and σS is the mean and standard deviation of S. CovSŜ is the Covariance of S and Ŝ. 1 = 0.01
  and 2 = 0.02 are two constant used to avoid zero division. Similarity is measured within the K 7*7 blocks divided from S and Ŝ.
  SSIM(S, Ŝ) = ΣKk=1( (2µSkµŜk + 1)(2CovSkŜk + 2)
  (µ2Sk + µ 2 Ŝk + 1)(σ2Sk + σ 2 Ŝk
  + 2) ) (28)
  Scale-invariant signal to noise ratio SiSNR (Le Roux et al., 2019) is widely used in speech restoration literatures to compare the energy of a signal to its background noise. SiSNR is calculated between target waveform s and waveform estimation ŝ:
  SiSNR(s, ŝ) = 10 ∗ log10 ‖ŝtarget‖2
  ‖enoise‖2 , (29)
  where ŝtarget = <ŝ,s>s‖ŝ‖2 . A higher SiSNR indicates less discrepancy between the estimation and target.
  Scale-invariant spectrogram to noise ratio SiSPNR is a spectral metric similar to SiSNR. They have the similar idea except SiSPNR is computed on the magnitude spectrogram. Given the target spectrogram S and estimation Ŝ, the computation of SiSPNR can be formulated as
  SiSPNR(S, Ŝ) = 10 ∗ log10 ‖Ŝtarget‖2
  ‖Enoise‖2 (30)
  where Ŝtarget = <Ŝ,S>S‖Ŝ‖2 . The scale invariant is guranteed by mean normalization of estimated and target spectrogram.
sections.heading: D APPENDIX D
sections.heading: D.1 EVALUATION RESULTS
sections.heading: D.2 ANALYSIS STAGE PERFORMANCE
sections.text: In this section, we report the mel spectrogram restoration score on different test sets. They are used to evaluate the performane of the analysis stage. We calculate the LSD, SiSPNR, and SSIM values.
  The Unprocessed column is calculated using the target and unprocessed mel spectrogram. And the Oracle-Mel column is calculated using the target spectrogram and itself.
  Table 10 shows that on DENOISE, DEREV, and ALL-GSR, all four VoiceFixer based models are effective on the restoration of mel spectrogram. Among the four analysis stage models, UNet is consistently better than the other three models.
  Table 11 lists the mel restoration performance on different sampling rates. Although VF-BiGRU has fewer parameters than VF-UNet, it still achieved the highest score on LSD and SiSPNR averagely. This result indicates that the recurrent structure might be more suitable for the mel spectrogram super-resolution task when the initial sampling rate is high.
sections.heading: D.3 DEMOS
sections.text: In this section, we provide some restoration demos using our proposed VoiceFixer. In Figure 12, we provide eight restoration demos using our VF-UNet model. All the audios used in our demo are either collected from the internet or recorded by ourselves. In each example, the left is the unprocessed spectrogram, and the right is the restored one. After restoration, these seriously distorted speech signals can be revert to relatively high quality.
  Figure 12b is the speech we recorded using Adobe Audition. We set the sampling rate to 8 kHz and manually add the clipping effect. It also contains some low-frequency noise and reverberation introduced by the recording device and environment. Figure 12a is a speech delivered by Amelia Earhart3, 1897-1937, appeared in the Library of Congress, United States. The original version sounds like a mumble. Figure 12f comes from an interview in a TV news program, which includes multiple distortions. Figure 12e is collected from the audio uploaded by a Youtuber4. Probably due to the recording device, her speech is deteriorated seriously by noise, and the energy of speech in the low-frequency part is also relatively low. Figure 12c is the restoration of a Chinese famous old movie railroad guerrilla5, which speech has limited bandwidth. The audio in Figure 12d is selected from a well-known TV series in China, romance of the three kindoms6, in which some parts of the spectrogram are masked off due to the previous audio compression. Figure 12g is a recording7 selected from a speech delivered by Sun Yat-sen, 1866-1925, which is in extremely lowresolution and includes multiple unknown distortions. Figure 12h shows the result of a subway broadcasting we recorded in Shanghai. The low-frequency part of speech almost lost completely, and the reverberation is serious.
  To sum up, all these examples showcase the effectiveness of the VoiceFixer model on GSR. And to our surprise, it can generate will on unseen distortions such as the spectrogram lost in Figure 12c, Figure 12f, and Figure 12d. In addition, Figure 12e shows that VoiceFixer is effective for the compensation of low-frequency energy, making speech sound less machinery and distant. Last but not least, despite the abnormal harmonic structure in the low-frequency part in Figure 12g, our proposed model can still restore its sound, which display the advantages of utilizing the prior knowledge of vocoder.
  3https://www.loc.gov/item/afccal000004 4https://v.ixigua.com/egVW74E/ 5https://www.youtube.com/watch?v=R8lY1qn2CHA 6https://www.youtube.com/watch?v=6h7N4Cl1lTw 7https://www.bilibili.com/video/BV1WW411V7fR
title: VOICEFIXER: TOWARD GENERAL SPEECH RESTORA-