summaries: The paper proposes an approach called VoiceFixer which is aimed at restoring degraded speech signals. The paper considers a variety of speech degradations - additive noise, reverberations, clipping and limited bandwidth. The paper describes a two stage approach in which the first stage aims to produce restored mel-spectrogram and then a vocoder is used to synthesize the  speech from the restored mel-spectrogram. Experiments are done using the VCTK dataset and experiments are done using single distortions as well as combinations of all 4 distortions.

summaries: This paper proposes a general speech restoration (GSR) task that tries to remove multiple distortions in a single model. In addition, it also presents a generative framework called VoiceFixer consisting of analysis and synthesis stages to address the general speech restoration task. In VoiceFixer, the authors employ a ResNet for modeling the analysis stage and a TFGAN-based neural vocoder for synthesis stage. They report that their baseline GSR and VoiceFixer surpass the single speech restoration (SSR) models with more improved results by the latter. Their idea was well described and the experiments are systematical and extensive. The results are consistent and clear.  The contribution of this paper is to incorporate a variety of speech restoration tasks including speech denoising, super-resolution, dereverberation, declipping, etc. in a single unified task called GSR. Another is the proposal of a well-performing generative speech restoration framework called VoiceFixer.

summaries: This paper introduces a unified view of several speech restoration problems including denoising, decliping, dereverberation and audio super-resolution. In order to solve the problem of the general speech restoration task, the authors propose a U-Net architecture which is trained on all of these tasks simultaneously during training time. The authors conduct extensive experiments for the general speech restoration task as well as the individual tasks where they compare the proposed models with more specialized models bounded to each distortion. The experimental results show that the proposed VoiceFixer combination of the model and the analysis-synthesis procedure are capable of effectively removing the speech distortions and in some cases outperform previous approaches in the literature.

summaries: The paper proposes a single system to deal with the speech enhancement tasks of denoising, dereverb, bandwidth extension (BWE) and declipping. The system is a two-stage system composed of an analysis module producing mel-band masks and a synthesis module using a vocoder. Both modules reuse existing architectures. Results comparing the proposed system to author-derived counterparts of it and to some existing systems for specific tasks show improvement for the proposed system in some cases, while achieving similar performance as existing systems in other cases. Data and code for reproducibility are provided.

abstractText: Speech restoration aims to remove distortions in speech signals. Prior methods mainly focus on single-task speech restoration (SSR), such as speech denoising or speech declipping. However, SSR systems do not address the general speech restoration problem and show limited performance in some speech restoration tasks such as speech super-resolution. To overcome those limitations, we propose a general speech restoration (GSR) task that attempts to remove multiple distortions simultaneously. Furthermore, we propose VoiceFixer1, a generative framework to address the GSR task. VoiceFixer consists of an analysis stage and a synthesis stage to mimic the speech analysis and comprehension of the human auditory system. We employ a ResUNet to model the analysis stage and a neural vocoder to model the synthesis stage. We evaluate VoiceFixer with additive noise, room reverberation, low-resolution, and clipping distortions. Our baseline GSR model achieves a 0.499 higher mean opinion score (MOS) than the speech denoising SSR model. VoiceFixer further surpasses the GSR baseline model on the MOS score by 0.256. Moreover, we observe that VoiceFixer generalizes well to severely degraded real speech recordings, indicating its potential in restoring old movies and historical speeches.

authors:
Fanhu Bie
Dong Wang
Jun Wang
Thomas Fang Zheng

title: Detection and reconstruction of clipped speech for speaker recognition

venue: Speech Communication,

authors:
Sawyer Birnbaum
Volodymyr Kuleshov
Zayd Enam
Pang Wei Koh
Stefano Ermon

title: Temporal film: Capturing long-range sequence dependencies with feature-wise modulations

title: Auditory scene analysis: The perceptual organization of sound

venue: MIT press,

authors:
Benjamin Cauchi
Ina Kodrasi
Robert Rehr
Stephan Gerlach
Ante Jukic
Timo Gerkmann
Simon Doclo
Stefan Goetze

title: Joint dereverberation and noise reduction using beamforming and a single-channel speech enhancement scheme

venue: In Proc. REVERB Challenge Workshop,

authors:
Junyoung Chung
Caglar Gulcehre
KyungHyun Cho
Yoshua Bengio

title: Empirical evaluation of gated recurrent neural networks on sequence modeling

venue: arXiv preprint arXiv:1412.3555,

authors:
Djork-Arné Clevert
Thomas Unterthiner
Sepp Hochreiter

title: Fast and accurate deep network learning by exponential linear units

venue: arXiv preprint arXiv:1511.07289,

authors:
Ross Cutler
Ando Saabas
Tanel Parnamaa
Markus Loide
Sten Sootla
Marju Purin
Hannes Gamper
Sebastian Braun
Karsten Sorensen
Robert Aichner

title: Acoustic echo cancellation challenge

venue: In INTERSPEECH,

authors:
Alexandre Defossez
Gabriel Synnaeve
Yossi Adi

title: Real time speech enhancement in the waveform domain

venue: arXiv preprint arXiv:2006.12847,

authors:
Adoram Erell
Mitch Weintraub

title: Estimation using log-spectral-distance criterion for noise-robust speech recognition

venue: In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing,

authors:
Simon Godsill
Peter Rayner
Olivier Cappé

title: Digital audio restoration

venue: In Applications of Digital Signal Processing to Audio and Acoustics,

authors:
Timothy D Griffiths
Jason D Warren

title: The planum temporale as a computational hub

venue: Trends in Neurosciences,

authors:
Archit Gupta
Brendan Shillingford
Yannis Assael
Thomas C Walters

title: Speech bandwidth extension with wavenet

venue: In IEEE Workshop on Applications of Signal Processing to Audio and Acoustics,

authors:
Kun Han
Yuxuan Wang
DeLiang Wang
William S Woods
Ivo Merks
Tao Zhang

title: Learning spectral mapping for speech dereverberation and denoising

venue: IEEE/ACM Transactions on Audio, Speech, and Language Processing,

authors:
Xiang Hao
Xiangdong Su
Shixue Wen
Zhiyu Wang
Yiqian Pan
Feilong Bao
Wei Chen

title: Masking and inpainting: A two-stage speech enhancement approach for low snr and nonstationary noise

authors:
Yanxin Hu
Yun Liu
Shubo Lv
Mengtao Xing
Shimin Zhang
Yihui Fu
Jian Wu
Bihong Zhang
Lei Xie

title: DCCRN: Deep complex convolution recurrent network for phase-aware speech enhancement

authors:
Kevin Jarrett
Koray Kavukcuoglu
Marc’Aurelio Ranzato
Yann LeCun

title: What is the best multistage architecture for object recognition

venue: In International Conference on Computer Vision, pp

authors:
Lauri Juvela
Bajibabu Bollepalli
Junichi Yamagishi
Paavo Alku

title: GELP: GAN-Excited linear prediction for speech synthesis from mel-spectrogram

title: Lectures on Wiener and Kalman filtering

venue: In Lectures on Wiener and Kalman Filtering,

authors:
Hamidreza Baradaran Kashani
Ata Jodeiri
Mohammad Mohsen Goodarzi
Shabnam Gholamdokht Firooz

title: Image to image translation based on convolutional neural network approach for speech declipping

title: Neural and cognitive mechanisms affecting perceptual adaptation to distorted speech

venue: PhD thesis,

authors:
Keisuke Kinoshita
Marc Delcroix
Takuya Yoshioka
Tomohiro Nakatani
Emanuel Habets
Reinhold Haeb-Umbach
Volker Leutnant
Armin Sehr
Walter Kellermann
Roland Maas

title: The REVERB challenge: A common evaluation framework for dereverberation and recognition of reverberant speech

authors:
Sran Kitić
Nancy Bertin
Rémi Gribonval

title: Sparsity and cosparsity for audio declipping: a flexible non-convex approach

venue: In International Conference on Latent Variable Analysis and Signal Separation,

authors:
Jungil Kong
Jaehyeon Kim
Jaekyoung Bae

title: HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis

venue: arXiv preprint arXiv:2010.05646,

authors:
Qiuqiang Kong
Yong Xu
Iwona Sobieraj
Wenwu Wang
Mark D Plumbley

title: Sound event detection and time–frequency segmentation from weakly labelled data

authors:
Qiuqiang Kong
Yin Cao
Haohe Liu
Keunwoo Choi
Yuxuan Wang

title: Decoupling magnitude and phase estimation with deep resunet for music source separation

venue: In The International Society for Music Information Retrieval,

authors:
Qiuqiang Kong
Haohe Liu
Xingjian Du
Li Chen
Rui Xia
Yuxuan Wang

title: Speech enhancement with weakly labelled data from audioset

venue: arXiv preprint arXiv:2102.09971,

authors:
Juho Kontio
Laura Laaksonen
Paavo Alku

title: Neural network-based artificial bandwidth expansion of speech

authors:
Volodymyr Kuleshov
S Zayd Enam
Stefano Ermon

title: Audio super resolution using neural networks

venue: arXiv preprint arXiv:1708.00853,

authors:
Kundan Kumar
Rithesh Kumar
Thibault de Boissiere
Lucas Gestin
Wei Zhen Teoh
Jose Sotelo
Alexandre de Brébisson
Yoshua Bengio
Aaron Courville

title: Melgan: Generative adversarial networks for conditional waveform synthesis

venue: arXiv preprint arXiv:1910.06711,

authors:
Rithesh Kumar
Kundan Kumar
Vicki Anand
Yoshua Bengio
Aaron Courville

title: NU-GAN: High resolution neural upsampling with gan

venue: arXiv preprint arXiv:2010.11362,

authors:
Frances Y Kuo
Ian H Sloan

title: Lifting the curse of dimensionality

venue: Notices of the American Mathematical Society,

authors:
Jonathan Le Roux
Scott Wisdom
Hakan Erdogan
John R Hershey

title: SDR–half-baked or well done

authors:
Junhyeok Lee
Seungu Han

title: Nu-wave: A diffusion probabilistic model for neural audio upsampling

venue: arXiv preprint arXiv:2104.02321,

authors:
Kehuang Li
Chin-Hui Lee

title: A deep neural network approach to speech bandwidth expansion

authors:
Yunpeng Li
Marco Tagliasacchi
Oleg Rybakov
Victor Ungureanu
Dominik Roblek

title: Real-time speech frequency bandwidth extension

authors:
Teck Yian Lim
Raymond A Yeh
Yijia Xu
Minh N Do
Mark Hasegawa-Johnson

title: Timefrequency networks for audio super-resolution

authors:
Ju Lin
Yun Wang
Kaustubh Kalgaonkar
Gil Keren
Didi Zhang
Christian Fuegen

title: A two-stage approach to speech bandwidth extension

authors:
Haohe Liu
Lei Xie
Jian Wu
Geng Yang

title: Channel-wise subband input for better voice and accompaniment separation on high resolution music

venue: arXiv preprint arXiv:2008.05216,

title: Speech enhancement: theory and practice

venue: CRC press,

authors:
Yi Luo
Nima Mesgarani

title: Conv-tasnet: Surpassing ideal time–frequency magnitude masking for speech separation

venue: IEEE/ACM Transactions on Acoustics, Speech, and Signal Processing,

authors:
Craig Macartney
Tillman Weyde

title: Improved speech enhancement with the Wave-U-Net

venue: arXiv preprint arXiv:1811.11307,

authors:
Wolfgang Mack
Emanuël AP Habets

title: Declipping speech using deep filtering

title: Spectral subtraction based on minimum statistics

authors:
Annamaria Mesaros
Toni Heittola
Tuomas Virtanen

title: A multi-device dataset for urban acoustic scene classification

venue: arXiv preprint arXiv:1807.09840,

authors:
Masanori Morise
Fumiya Yokomori
Kenji Ozawa

title: World: a vocoder-based high-quality speech synthesis system for real-time applications

venue: IEICE Transactions on Information and Systems,

authors:
Anna K Nábělek
Tomasz R Letowski
Frances M Tucker

title: Reverberant overlap-and self-masking in consonant identification

venue: The Journal of the Acoustical Society of America,

authors:
Yoshihisa Nakatoh
Mineo Tsushima
Takeshi Norimatsu

title: Generation of broadband speech from narrowband speech based on linear mapping

venue: Electronics and Communications in Japan,

authors:
Arun Narayanan
DeLiang Wang

title: Ideal ratio mask estimation using deep neural networks for robust speech recognition

authors:
Patrick A Naylor
Nikolay D Gaubitch

title: Speech dereverberation

venue: Springer Science & Business Media,

authors:
Aaron van den Oord
Sander Dieleman
Heiga Zen
Karen Simonyan
Oriol Vinyals
Alex Graves
Nal Kalchbrenner
Andrew Senior
Koray Kavukcuoglu

title: Wavenet: A generative model for raw audio

venue: arXiv preprint arXiv:1609.03499,

authors:
Santiago Pascual
Antonio Bonafonte
Joan Serra

title: SEGAN: Speech enhancement generative adversarial network

venue: arXiv preprint arXiv:1703.09452,

authors:
Wei Ping
Kainan Peng
Kexin Zhao
Zhao Song

title: Waveflow: A compact flow-based model for raw audio

venue: In International Conference on Machine Learning,

authors:
Adam Polyak
Lior Wolf
Yossi Adi
Ori Kabeli
Yaniv Taigman

title: High fidelity speech regeneration with application to speech enhancement

authors:
Ryan Prenger
Rafael Valle
Bryan Catanzaro

title: Waveglow: A flow-based generative network for speech synthesis

authors:
Yi Ren
Yangjun Ruan
Xu Tan
Tao Qin
Sheng Zhao
Zhou Zhao
Tie-Yan Liu

title: FastSpeech: Fast, robust and controllable text to speech

authors:
Lucas Rencker
Francis Bach
Wenwu Wang
Mark D Plumbley

title: Sparse recovery and dictionary learning from nonlinear compressive measurements

venue: IEEE Transactions on Signal Processing,

authors:
Dayana Ribas
Emmanuel Vincent
José Ramón Calvo

title: A study of speech distortion conditions in real scenarios for speech processing applications

venue: In IEEE Spoken Language Technology Workshop,

authors:
Antony W Rix
John G Beerends
Michael P Hollier
Andries P Hekstra

title: Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs

authors:
Olaf Ronneberger
Philipp Fischer
Thomas Brox

title: U-net: Convolutional networks for biomedical image segmentation

venue: In International Conference on Medical image computing and computerassisted intervention,

title: Speech enhancement based on a priori signal to noise estimation

venue: In Proceedings of the IEEE Conference on Acoustics, Speech, and Signal Processing Conference Proceedings,

authors:
Boaz Schwartz
Sharon Gannot
Emanuël AP Habets

title: Online speech dereverberation using kalman filter and em algorithm

authors:
Jonathan Shen
Ruoming Pang
Ron J Weiss
Mike Schuster
Navdeep Jaitly
Zongheng Yang
Zhifeng Chen
Yu Zhang
Yuxuan Wang
Rj Skerrv-Ryan

title: Natural TTS synthesis by conditioning wavenet on Mel spectrogram predictions

authors:
Yao Shi
Hui Bu
Xin Xu
Shaoji Zhang
Ming Li

title: Aishell-3: A multi-speaker mandarin tts corpus and the baselines

venue: arXiv preprint arXiv:2010.11567,

authors:
Xiaofeng Shu
Yehang Zhu
Yanjie Chen
Li Chen
Haohe Liu
Chuanzeng Huang
Yuxuan Wang

title: Joint echo cancellation and noise suppression based on cascaded magnitude and complex mask estimation

venue: arXiv preprint arXiv:2107.09298,

authors:
Serkan Sulun
Matthew EP Davies

title: On filter generalization for music bandwidth extension using deep neural networks

venue: IEEE Journal of Selected Topics in Signal Processing,

authors:
Ke Tan
Yong Xu
Shi-Xiong Zhang
Meng Yu
Dong Yu

title: Audio-visual speech separation and dereverberation with a two-stage multimodal network

authors:
Qiao Tian
Yi Chen
Zewang Zhang
Heng Lu
Linghui Chen
Lei Xie
Shan Liu

title: TFGAN: Time and frequency domain based generative adversarial network for high-fidelity speech synthesis

title: Noisy speech database for training speech enhancement algorithms and TTS models

authors:
Tim Van den Bogaert
Simon Doclo
Jan Wouters
Marc Moonen

title: Speech enhancement with multichannel wiener filter techniques in multimicrophone binaural hearing aids

title: Audio analysis and spectral restoration workflows using adobe audition

venue: In Audio Engineering Society Conference: 33rd International Conference: Audio Forensics-Theory and Practice,

authors:
Emmanuel Vincent
Shinji Watanabe
Aditya Arie Nugraha
Jon Barker
Ricard Marxer

title: An analysis of environment, microphone and data simulation mismatches in robust speech recognition

venue: Computer Speech & Language,

authors:
Heming Wang
DeLiang Wang

title: Towards robust speech super-resolution

authors:
Hong Wang
Fumitada Itakura

title: Dereverberation of speech signals based on sub-band envelope estimation

venue: IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences,

authors:
Wenfu Wang
Shuang Xu
Bo Xu

title: First step towards end-to-end parametric tts synthesis: Generating spectral parameters with neural attention

authors:
Zhou Wang
Alan C Bovik
Hamid R Sheikh
Eero P Simoncelli

title: Image quality assessment: from error visibility to structural similarity

venue: IEEE Transactions on Image Processing,

authors:
Donald S Williamson
DeLiang Wang

title: Time-frequency masking in the complex domain for speech dereverberation and denoising

authors:
Junichi Yamagishi
Christophe Veaux
Kirsten MacDonald

title: Cstr vctk corpus: English multispeaker corpus for cstr voice cloning toolkit

authors:
Ryuichi Yamamoto
Eunwoo Song
Jae-Min Kim

title: Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram

authors:
Chengzhu Yu
Heng Lu
Na Hu
Meng Yu
Chao Weng
Kun Xu
Peng Liu
Deyi Tuo
Shiyin Kang
Guangzhi Lei

title: Durian: Duration informed attention network for multimodal synthesis

venue: arXiv preprint arXiv:1909.01700,

authors:
Pavel Záviška
Pavel Rajmic
Ondřej Mokrỳ
Zdeněk

title: Prŭša. A proper version of synthesis-based sparse audio declipper

authors:
Pavel Záviška
Pavel Rajmic
Jiřı́ Schimmel

title: Psychoacoustically motivated audio declipping based on weighted l 1 minimization

venue: 42nd International Conference on Telecommunications and Signal Processing,

authors:
Pavel Záviška
Pavel Rajmic
Alexey Ozerov
Lucas Rencker

title: A survey and an extensive evaluation of popular audio declipping methods

authors:
Yan Zhao
Zhong-Qiu Wang
DeLiang Wang

title: Two-stage deep learning for noisy-reverberant speech enhancement

title: 2019) treat the declipping as an image-toimage translation problem and utilize the UNet to perform spectral mapping. Currently, most of the state-of-the-art methods are unsupervised (Záviška et al., 2020

venue: neural network (DNN) (Bie et al.,

heading: 1 INTRODUCTION

text:
Speech restoration is a process to restore degraded speech signals to high-quality speech signals. Speech restoration is an important research topic since distortions are ubiquitous. For example, speech is usually surrounded by background noise, blurred by room reverberations, or recorded by low-quality devices (Godsill et al., 2002). Those distortions degrade the perceptual quality of speech for human listeners. Speech restoration has a wide range of applications such as online meeting (Defossez et al., 2020), hearing aids (Van den Bogaert et al., 2009), and audio editting (Van Winkle, 2008). Still, speech restoration remains a challenging problem due to the large variety of distortions in the world.
Previous works in speech restoration mainly focus on single task speech restoration (SSR), which deals with only one type of distortion at a time. For example, speech denoising (Loizou, 2007), speech dereveberation (Naylor & Gaubitch, 2010), speech super-resolution (Kuleshov et al., 2017), or speech declipping (Záviška et al., 2020). However, in the real world, speech signal can be degraded by several different distortions simultaneously, which means previous SSR systems oversimplify the speech distortion types (Kashani et al., 2019; Lin et al., 2021; Kuleshov et al., 2017; Birnbaum et al., 2019). The mismatch between the training data used in SSR and the testing data from the real world degrades the speech restoration performance.
To address the mismatch problem, we propose a new task called general speech restoration (GSR), which aims at restoring multiple distortions in a single model. A numerous studies (Cutler et al., 2021; Cauchi et al., 2014; Han et al., 2015) have reported the benefits of jointly training multiple speech restoration tasks. Nevertheless, performing GSR using one-stage systems still suffer from the problems in each SSR task. For example, on generative tasks such as speech superresolution (Kuleshov et al., 2017), one-stage models tend to overfitting the filter (Sulun & Davies, 2020) used during training. Based on these observations, we propose a two-stage system called VoiceFixer.
1Restoration samples: https://anonymous20211004.github.io/demo-vf/
The design of VoiceFixer is motivated by the biological mechanisms of human hearing when restoring distorted speech. Intuitively, if a person tries to identify a strongly distorted voice, his/her brain can do the recovery by utilizing both the distorted speech signal and the prior knowledge of the language. As shown in Figure 1, the speech distortion perception is modeled by neuroscientists as a two-stage process, including an auditory scene analysis stage (Bregman, 1994), and a high level comprehension/synthesis stage (Griffiths & Warren, 2002). In the analysis stage, the sound information is first transformed into acoustic features by primary auditory cortex (PAC). Then planum temporale (PT), the cortical area posterior to the auditory cortex, acts as a computational hub by segregating and matching the acoustic features to low level spectrotemporal representations. In the synthesis stage, a high order cortical area is hypothesised to perform the high level perception tasks (Griffiths & Warren, 2002; Kennedy-Higgins, 2019). Our proposed VoiceFixer systems model the analysis stage with spectral transformations and a deep residual UNet, and the synthesis stage with a convolutional vocoder trained using adversarial losses. One advantage of the two-stage VoiceFixer is that the analysis and synthesis stages can be trained separately. Two-stage methods have also been successfully applied to the speech synthesis task (Wang et al., 2016; Ren et al., 2019) where acoustic models and vocoders are trained separately.
VoiceFixer is the first GSR model that is able to restore a wide range of low-resolution speech sampled from 2 kHz to 44.1 kHz, which is different from previous studies working on constant sampling rates (Lim et al., 2018; Wang & Wang, 2021; Lee & Han, 2021). To the best of our knowledge, VoiceFixer is the first model that jointly performs speech denoising, speech dereverberation, speech super-resolution, and speech declipping in a unified model.
The rest of this paper is organized as follows. Section 2 introduces the formulations of speech distortions. Section 3 describes the design of VoiceFixer. Section 4 discusses the evaluation results. Appendixes introduce related works and show speech restoration demos.

heading: 2 PROBLEM FORMULATIONS

text:
We denote a segment of a speech signal as s ∈ RL, where L is the samples number in the segment. We model the distortion process of the speech signal as a function d(·). The degraded speech x ∈ RL can be written as:
x = d(s). (1)
Speech restoration is a task to restore high-quality speech ŝ from x:
ŝ = f(x) (2)
where f(·) is the restoration function and can be viewed as a reverse process of d(·). The target is to estimate s by restoring ŝ from the observed speech x. Recently, several deep learning based one-stage methods have been proposed to model f(·) such as fully connected neural networks, recurrent neural networks, and convolutional neural networks. Detailed introductions can be found in Appendix A.
Distortion modeling is an important step to simulate distorted speech when building speech restoration systems. Several previous works model distortions in a sequential order (Vincent et al., 2017; Tan et al., 2020; Zhao et al., 2019). Similarly, we model the distortion d(·) as a composite function:
d(x) = d1 ◦ d2 ◦ ...dQ(x), dq ∈ D, q = 1, 2, ..., Q, (3)
where ◦ stands for function composition and Q is the number of distortions to consist d(·). Set D = {dv(·)}Vv=1 is the set of distortion types where V is the total number of types. Equation 3 describes the procedure of compounding different distortions from D in a sequential order. We introduce four speech distortions as follows.
Additive noise is one of the most common distortion and can be modeled by the addition between speech s and noise n ∈ RL:
dnoise(s) = s + n. (4)
Reverberation is caused by the reflections of signal in a room. Reverberation makes speech signals sound distant and blurred. It can be modeled by convolving speech signals with a room impulse response filter (RIR) r:
drev(s) = s ∗ r, (5)
where ∗ stands for convolution operation. Low-resolution distortions refer to audio recordings that are recorded in low sampling rates or with limited bandwidth. There are many causes for low-resolution distortions. For example, when microphones have low responses in high-frequency, or audio recordings are compressed to low sampling rates, the high frequencies information will be lost. We follow the description in Wang & Wang (2021) to produce low-resolution distortions but add more filter types (Sulun & Davies, 2020). After designing a low pass filter h, we first convolve it with s to avoid the aliasing phenomenon. Then we perform resampling on the filtered result from the original sampling rate o to a lower sampling rate u:
dlow res(s) = Resample(s ∗ h, o, u), (6)
Clipping distortions refer to the clipped amplitude of audio signals, which are usually caused by low-quality microphones. Clipping can be modeled by restricting signal amplitudes within [−η,+η]:
dclip(s) = max(min(s, η),−η), η ∈ [0, 1]. (7)
In the frequency domain, the clipping effect produces harmonic components in the high-frequency part and degrades speech intelligibility accordingly.

heading: 3 METHODOLOGY

heading: 3.1 ONE-STAGE SPEECH RESTORATION MODELS

text:
Previous deep learning based speech restoration models are usually in one stage. That is, a model predicts restored speech ŝ from input x directly:
f : x→ ŝ. (8)
The mapping function f(·) can be modeled by time domain speech restoration systems such as onedimensional convolutional neural networks (Luo & Mesgarani, 2019) or frequency domain systems such as mask-based (Narayanan & Wang, 2013) methods:
Ŝ = (Fsp(|X| ; θ) (|X|+ ))ej∠X . (9)
where X is the short-time fourier transform (STFT) of x and is a small positive constant. X has a shape of T×F where T is the number of frame and F is the number of frequency bins. The output of the mask estimation function F (·; θ) is multiplied by the magnitude spectrogram |X| to produce the target spectrogram estimation Ŝ. Then, inverse short-time fourier transform (iSTFT) is applied on Ŝ to obtain ŝ. The one-stage speech restoration models are typically optimized by minimizing the mean absolute error (MAE) loss between the estimated spectrogram Ŝ and the target spectrogram S:
L = ∥∥∥|Ŝ| − |S|∥∥∥
1 (10)
Previous one-stage models usually build on high-dimensional features such as time samples and the STFT spectrograms. However, Kuo & Sloan (2005) point out that the high-dimensional features will lead to exponential growth in search space. The model can work on the high-dimensional features under the premise of enlarging the model capacity but may also fail in challenging tasks. Therefore, it would be beneficial if we could build a system on more delicate low-dimensional features.

heading: 3.2 VOICEFIXER

text:
In this study, we propose VoiceFixer, a two-stage speech restoration framework. Multi-stage methods have achieved state-of-the-art performance in many speech processing tasks (Jarrett et al., 2009; Tan et al., 2020). In speech restoration, our proposed VoiceFixer breaks the conventional one-stage system into a two-stage system:
f : x 7→ z, (11)
g : z 7→ ŝ. (12) Equation 11 denotes the analysis stage of VoiceFixer where a distorted speech x is mapped into a representation z. Equation 12 denotes the synthesis stage of VoiceFixer, which synthesize z to the restored speech ŝ. Through the two-stage processing, VoiceFixer mimics the human perception of speech described in Section 1.

heading: 3.2.1 ANALYSIS STAGE

text:
The goal of the analysis stage is to predict the intermediate representation z, which can be used later to recover the speech signal. In our study, we choose mel spectrogram as the intermediate representation. Mel spectrogram has been widely used in many speech proceessing tasks (Shen et al., 2018; Kong et al., 2019; Narayanan & Wang, 2013). The frequency dimension of mel spectrogram is usually much smaller than that of STFT thus can be regarded as a way of feature dimension
reduction. So, the objective of the analysis stage becomes to restore mel spectrograms of the target signals. The mel spectrogram restoration process can be written as the following equation,
Ŝmel = fmel(Xmel;α) (Xmel + ), (13)
where Xmel is the mel spectrogram of x. It is calculated by Xmel = |X|W where W is a set of mel filter banks with shape of F × F ′. The columns of W are not divided by the width of their mel bands, i.e., area normalization, because this will make the restoration model difficult to recover the high-frequency part. The mapping function fmel(·;α) is the mel restoration mask estimation model parameterized by α. The output of fmel is multiplied by Xmel to predict the target mel spectrogram.
We use ResUNet (Kong et al., 2021a) to model the analysis stage as shown in Figure 3a, which is an improved UNet (Ronneberger et al., 2015). The ResUNet consists of several encoder and decoder blocks. There are skip connections between encoder and decoder blocks at the same level. As is shown in Figure 3b, both encoder and decoder block share the same structure, which is a series of residual convolutions (ResConv). Each convolutional layer in ResConv consists of a batch normalization (BN), a leakyReLU activation, and a linear convolutional operation. The encoder blocks apply average pooling for downsampling. The decoder blocks apply transpose convolution for upsampling. In addition to ResUNet, we implement the analysis stage with fully connected deep neural network (DNN), and bidirectional gated recurrent units (BiGRU) (Chung et al., 2014) for comparison. The DNN consists of six fully connected layers. The BiGRU has similar structures with DNN except for replacing the last two layers of DNN into bi-directional GRU layers.
The details of these three models are discussed in Appendix B.1. We will refer to ResUNet as UNet later for abbreviation. We optimize the model in the analysis stage using the MAE loss between the estimated mel spectrogram Ŝmel and the target mel spectrogram Smel:
Lana = ∥∥∥Ŝmel − Smel∥∥∥
1 (14)

heading: 3.2.2 SYNTHESIS STAGE

text:
The synthesis stage is realized by a neural vocoder that synthesizes the mel spectrogram into waveform as denoted in Equation 15:
ŝ = g(Xmel;β), (15)
where g(·;β) stands for the vocoder model parameterized by β. We employ a recently proposed nonautoregressive model, time and frequency domain based generative adversarial network (TFGAN), as the vocoder.
Figure 4 shows the detailed architecture of TFGAN, in which the input mel spectrogram Xmel will first pass through a condition network CondNet, which contains N1 one-dimensional convolution layers with exponential linear unit activations (Clevert et al., 2015). Then, in UpNet, the feature is upsampledN2 times with ratios of s0,s1,..., and sN2−1 using UpsampleBlock and ResStacks. Within the UpsampleBlock, the input is first passed through a leakyReLU activation and then fed into a sinusoidal function, which output is added to its input to remove periodic artifacts in breathing part of speech. Then, the output is bifurcated into two branches for upsampling. One branch repeats the samples sn times followed by a one-dimensional convolution. The other branch uses a stride sn transpose convolution. The output of the repeat and transpose convolution branches are added together as the output of UpsampleBlock. ResStacks module contains two dilated convolution layers with leakyReLU activations. The exponentially growing dilation in ResStack enable the model to capture long range dependencies. The TFGAN in our synthesis model applies N2 = 4. After four UpsampleBlock blocks with ratios [7, 7, 3, 3], each frame of the mel spectrogram is transformed into a sequence with 441 samples corresponding to 10 ms of audio sampled at 44.1 kHz.
The training criteria of the vocoder consist of frequency domain loss LF , time domain loss LT , and weighted discriminator loss LD:
Lsyn = LF + LT + λDLD, (16)
The frequency domain loss LF is the combination of a mel loss Lmel and multi-resolution spectrogram losses:
LF (ŝ, s) = λmelLmel(ŝ, s) + KF∑ k=1 (λscL (k) sc (ŝ, s) + λmagL (k) mag(ŝ, s)) (17)
where Lsc and Lmag are the spectrogram losses calculated in the linear and log scale, respectively. There are KF different window sizes ranging from 64 to 4096 to calculate Lsc and Lmag so that the trained vocoder is tolerant over phase mismatch (Yamamoto et al., 2020; Juvela et al., 2019). Table 2 in Appendix B.2 shows the detailed configurations.
Time domain loss is complementary to frequency domain loss to address problems such as periodic artifacts. Time domain loss combines segment loss L(k)seg , energy loss L (k) energy and phase loss L (k) phase:
LT (ŝ, s) = KT∑ k=1 (λsegL (k) seg (ŝ, s)) + λenergyL (k) energy(ŝ, s) + λphaseL (k) phase(ŝ, s) (18)
where segment loss L(k)seg , energy loss L (k) energy and phase loss L (k) phase are described in Equation 24, 25, and 26 of Appendix B.2. There are KT different window sizes ranging from 1 to 960 to calculate time domain loss at different resolutions. The details of window sizes are shown in Table 3 of Appendix B.2. The energy loss and phase loss have the advantage of alleviating metallic sounds.
Discriminative training is an effective way to train neural vocoders (Kong et al., 2020; Kumar et al., 2019). In our study, we utilize a group of discriminators, including a multi-resolution time discriminator DT , a subband discriminator Dsub, and frequency discriminator DF :
D(s) = RT∑ r=1 D (r) T (s) +Dsub(s) +DF (s) (19)
LD(s, ŝ) = min g max D
(Es(log(D(s))) + Eŝ(log(1−D(ŝ)))). (20)
The multi-resolution discriminators DT take signals from RT kinds of time resolutions after average pooling as input. The subband discriminator Dsub performs subband decomposition (Liu et al., 2020) on the waveform, producing four subband signals to feed into four T-discrminators, respectively. Frequency discriminator DF takes the linear spectrogram as input and outputs real or fake labels. The bottom part of Figure 4 shows the main idea of T-discriminator and F-discriminator. Appendix B.2 describes the detailed discriminator architectures.
There are two advantages of using neural vocoder in the synthesis stage. First, neural vocoder trained using a large amount of speech data contains prior knowledge on the structural distribution of speech signals, which is crucial to the restoration of distorted speech. The amount of training data of vocoder is more than that used in conventional SSR methods with limited speaker numbers. Second, the neural vocoder typically takes the mel spectrogram as input, resulting in fewer feature dimensions than the STFT features. The reduction in dimension helps to lower computational costs and achieve better performance in the analysis stage.

heading: 4 EXPERIMENTS

heading: 4.1 DATASETS AND EVALUATION METRICS

text:
Training sets: The training speech datasets we use including VCTK (Yamagishi et al., 2019), AISHELL-3 (Shi et al., 2020), and HQ-TTS. We call the noise datasets used for training as VDNoise. To simulate the reverberations, we employ a set of RIRs to create an RIR-44k dataset. We use VCTK, VD-Noise, and the training part of RIR-44k to train the analysis stage. AISHELL-3, VCTK, and HQ-TTS datasets are used to train the vocoder in the synthesis stage. The details of those datasets and the RIRs simulation configurations are discussed in Appendix C.1.
Test sets: We employ VCTK-Demand (Valentini-Botinhao et al., 2017) as the denoising test set and name it as DENOISE. We call our speech super-resolution, declipping, and dereverberation evaluation test sets as SR, DECLI, and DEREV, respectively. In addition, we create an ALL-GSR test set containing all distortions. We introduce the details of how we build these test sets in Appendix C.3.
Evaluation metrics: The metrics we adopt include log-spectral distance (LSD) (Erell & Weintraub, 1990), wide band perceptual evaluation of speech quality (PESQ-wb) (Rix et al., 2001), structural similarity (SSIM) (Wang et al., 2004), and scale-invariant signal to noise ratio (SiSNR) (Le Roux et al., 2019). We use mean opinion scores (MOS) to subjevtively evaluate different systems.
The output of the vocoder is not strictly aligned on sample level with the target, as is often the case in generative model (Kumar et al., 2020). This effect will degrade the metrics, especially for those calculated on time samples such as the SiSNR. So, to compensate the SiSNR, we design similar metric, scale-invariant spectrogram to noise ratio (SiSPNR), to measure the discrepancies on the spectrograms. Details of the metrics are described in Appendix C.4.

heading: 4.2 DISTORTIONS SIMULATION

text: For the SSR task, we perform only one type of distortion for evaluation. For the GSR task, we first assume that D = {dnoise, drev, dlow res, dclip} because those distortions are the most common distortions in daily environment (Ribas et al., 2016). Second, we assume that Q ≤ 4 in Equation 3. In other words, each distortion in D appears at most one time. Then, we generate the distortions following a specific order drev, dclip, dlow res, and dnoise. These distortions are added randomly using random configurations.

heading: 4.3 BASELINE SYSTEMS

text:
Table 5 in Appendix D summarizes all the experiments in this study. We implement several SSR and GSR systems using one-stage restoration models. For the GSR, we train a ResUNet model called GSR-UNet with all distortions. For the SSR models, we implement a Denoise-UNet for additive noise distortion, a Dereverb-UNet for reverberation distortion, a SR-UNet for low-resolution distortion, and a Declip-UNet for clipping distortion. For the SR task, we also include two stateof-the-art models, NuWave (Lee & Han, 2021) and SEANet (Li et al., 2021) for comparison. For declipping task, we compare with a state-of-the-art synthesis-based method SSPADE (Záviška et al.,
2019a). To explore the impact of model size of the mel restoration model, we setup ResUNets with two sizes, UNet-S and UNet, which have one and four ResConv blocks in each encoder and decoder block, respectively.

heading: 4.4 EVALUATION RESULTS

text:
Neural vocoder To evaluate the performance of the neural vocoder, we compare two baselines. The Target system denotes using the perfect s for evaluation. The Unprocessed system denotes using distorted speech x for evaluation. The Oracle-Mel system denotes using the mel spectrogram of s as input to the vocoder, which marks the performance of the vocoder. As shown in Table 1, the OracleMel system achieves a MOS score of 3.74, which is close to the Target MOS of 3.95, indicating that the vocoder performs well in the synthesis task.
General speech restorations: Table 1 shows the evaluation results on ALL-GSR test set. Figure 5 shows the box plot of the MOS scores of these systems. The GSR-UNet outperforms the two SSR models, Denoise-UNet and Dereverb-UNet by a large margin. It surpasses Denoise-UNet model by 0.5 on MOS score, which suggests the GSR model is more powerful than the SSR model on this test set. For convenience, we denote VoiceFixer as VF in tables and figures. We observe that the VF-UNet model achieves the highest MOS score and LSD score. Specifically, VF-UNet obtains 0.256 higher MOS score than that of GSR-UNet. This result indicates that VoiceFixer is better than ResUNet based one-stage model on overall quality. What’s more, we notice that the MOS score of VF-UNet is only 0.11 lower than the Oracle-Mel, demonstrating the good performance of the analysis stage. Among the VoiceFixer analysis models, the UNet front-end achieves the best. The VF-BiGRU model achieves similar subjective metrics with the VF-UNet model but has much lower MOS scores. This phenomenon shows that the improvement in subjective metrics in VoiceFixer is not always consistent with objective evaluation results.
Super-resolution: Table 6 in Appendix D.1 shows the evaluation results on the super-resolution test set SR. For the 2 kHz, 4 kHz, and 8 kHz to 44.1 kHz super-resolution tasks, VF-UNet achieves a significantly higher LSD, SiSPNR and SSIM scores than other models. The LSD value of VF-UNet in 2 kHz sampling rate is still higher than the 8 kHz sampling rate score of GSR-UNet, SR-UNet, NuWave, and SEANet. This demonstrates the strong performance of VoiceFixer on dealing with low sampling rate cases. The VF-BiGRU model outperforms VF-UNet-S model on average scores for its better performance on low upsample-ratio cases. MOS box plot in Figure 6b shows that VF-UNet performs the best on 8 kHz to 44.1 kHz test set. Figure 6e shows the MOS score of Unprocessed is close to Target on 24 kHz to 44.1 kHz test set, meaning limited perceptual difference between the two sampling rates. On this test set, SEANet even achieves a higher MOS score than Target. That’s due to the high-frequency part it generate contains more energy than that of Target, making the results sound clearer.
Denoising: We evaluate the speech denoising performance on the DENOISE test set and show results in Table 7 in Appendix D.1. We find that GSR-UNet preserves more details in the highfrequency part and has better PESQ and SiSPNR values than the denoising only SSR model DenoiseUNet. The reason might be that speech data augmentations and jointly performing super-resolution task can increase the generalization and inpainting ability of the model (Hao et al., 2020). The PESQ
score of VF-UNet reaches 2.43, higher than SEGAN, WaveUNet, and the model trained with weakly labeled data in Kong et al. (2021b). The MOS evaluations in Figure 6f on speech denoising task also demonstrate that the result of VF-UNet sound comparable with one-stage speech denoising models.
Declipping and dereveberation: Table 9 and Table 8 in Appendix D.1 show similar performance trends on the speech declipping and speech dereveberation. In both tasks, the SSR model DereverbUNet and Declip-UNet achieve the highest scores. The performance of GSR-UNet is slightly worse, but it is acceptable considering that GSR-UNet does not need extra training for each task. SSPADE performs better on SiSNR, but the PESQ and STOI scores are lower, especially in the 0.1 threshold case. The MOS score in Figure 6d shows that the clipping effect in the 0.25 threshold case is not easy to perceive, leading to high MOS scores across all methods. In Figure 6a, both Declip-UNet and VF-UNet achieve the highest objective scores on the 0.1 threshold clipping test set. On the dereverberation test set DEREV, VF-UNet achieves the highest MOS score 3.52.

heading: 5 CONCLUSIONS

text: In this work, we propose VoiceFixer, an effective approach for general speech restoration. The evaluation results show that VoiceFixer achieves leading performance across the general speech restoration, speech super-resolution, speech denoising, speech dereverberation, and speech declipping tasks. In the future, we will extend VoiceFixer to restore general audio signals, including music and general sounds.

heading: REPRODUCIBILITY STATEMENT

text: We make our code and datasets anonymously downloadable for painless reproducibility. Our pre-trained VoiceFixer and inference code are presented in https://github.com/ anonymous20211004/iclr2022-vf. The code for performing the experiments discussed in Section 4 is downloadable in https://github.com/anonymous20211004/ iclr2022-train-vf. The code can be used to conduct evaluations and generate reports on the metrics mentioned in Section 4.1 automatically. The NuWave is realized using the code open-sourced by Lee & Han (2021): https://github.com/mindslab-ai/nuwave. We reproduce SSPADE using the toolbox provided by Záviška et al. (2020) at https://rajmic.github.io/ declipping2020. Besides, we upload the training and testing sets mentioned in Section 4.1 to https://zenodo.org/record/5546723. The AISHELL-3 dataset is open-sourced at http://www.aishelltech.com/aishell_3. The HQ-TTS is a collection of datasets from openslr.org, as is described in Table 4 of Appendix C.1.

heading: A APPENDIX A

heading: A.1 SPEECH RESTORATION TASKS

text:
Speech super-resolution A lot of early studies (Nakatoh et al., 2002; Kontio et al., 2007) break super-resolution (SR) into spectral envelop estimation and excitation generation from the lowresolution part. At that time, the direct mapping from the low-resolution part to the high-resolution feature is not widely explored since the dimension of the high-resolution part is relatively high. Later, deep neural network (Li & Lee, 2015; Kuleshov et al., 2017) is introduced to perform SR. These approaches show better subjective quality comparing with traditional methods. To increase the modeling capacity, TFilm (Birnbaum et al., 2019) is proposed to model the affine transformation among each time block. Similarly, WaveNet also shows effectiveness in extending the bandwidth of a band-limited speech (Gupta et al., 2019). To utilize the information both from the time and frequency domain, Wang & Wang (2021) propose a time-frequency loss that can yield a balanced performance on different metrics. Recently, NU-GAN (Kumar et al., 2020) and NU-Wave (Lee & Han, 2021) pushed the target sample rate in SR to high fidelity, namely 48 kHz.
Although employing deep neural networks in BWE shows promising results, the generalization capability of these methods is still limited. For example, previous approaches (Kuleshov et al., 2017) usually train and test models with a fixed setup, i.e., fixing the initial and target sampling rates. However, in real-world applications, speech bandwidth is not usually constant. In addition, since the real high-low quality speech pair is hard to collect, typically, we produces low-quality audio with lowpass or bandpass filters during training. In this case, systems tend to overfit specific filters. As discussed in Sulun & Davies (2020), when the kind of filter used during training and testing differ, the performance can fall considerably. To alleviate filter overfitting, Sulun & Davies (2020) propose to train models with multiple kinds of lowpass filters, by which the unseen filters can be handled properly.
Speech declipping The methods for speech declipping can be categorized as supervised methods and unsupervised methods. The unsupervised, or blind methods usually perform declipping based on some generic regularization and assumption of what natural audio should look like, such as ASPADE (Kitić et al., 2015), dictionary learning (Rencker et al., 2019), and psychoacoustically motivated l1 minimization (Záviška et al., 2019b). The supervised models, mostly based on deep neural network (DNN) (Bie et al., 2015; Mack & Habets, 2019), are usually trained on clipped and unclipped data pairs. For example, Kashani et al. (2019) treat the declipping as an image-toimage translation problem and utilize the UNet to perform spectral mapping. Currently, most of the state-of-the-art methods are unsupervised (Záviška et al., 2020) because they are usually designed to work on different kinds of audio, while the supervised model mainly specialized on data similar to its training data. However, Záviška et al. (2020) believes supervised models still have the potential for better declipping performance.
Speech denoising Conventional methods are efficient and effective on dealing with stationary noise, such as spectral substraction (Martin, 1994) and Wiener and Kalman filtering (Kailath, 1981) By comparison, deep learning based models such as Conv-TasNet (Luo & Mesgarani, 2019) show higher subjective score and robustness on complex cases. Recently, new schemes have emerged for training speech denoising models. SEGAN (Pascual et al., 2017) tried a generative way. Kong et al. (2021b) achieved a denoising model using only weakly labeled data. And Polyak et al. (2021) realized a denoising model using a regeneration approach.
Speech dereveberation Some of the early methods in speech dereverberation, such as inverse filtering (Naylor & Gaubitch, 2010) and subband envelope estimation (Wang & Itakura, 1991), aiming at deconvolving the reverberant signal by estimating an inverse filter. However, the inverse filter is hard and not robust to estimate accurately. Other methods, like spectral substraction, is based on an overlap-masking (Nábělek et al., 1989) effect of reverberation. Schwartz et al. (2014) performed dereverberation using Kalman filter and expectation-maximization algorithm. Recently, deep learning based dereverberation methods have emerged as the state-of-the-art. Han et al. (2015) used a fully connected DNN to learn a spectral mapping from reverberant speech to clean speech. In Williamson & Wang (2017), similar to the masking-based denoising methods, they proposed to perform dereverberation using a time-frequency mask.

heading: A.1.1 JOINT RESTORATION AND SYNTHETIC RESTORATION

text:
Joint restoration Many works have adopted the joint restoration approach to improving models. To make the acoustic echo cancellation (AEC) result sound cleaner, MC-TCN (Shu et al., 2021) proposed to jointly perform AEC and noise suppression at the same time. MC-TCN achieved a mean opinion score of 4.41, outperforming the baseline of AEC Challenge (Cutler et al., 2021) by 0.54. Moreover, in the REVERB challenge (Kinoshita et al., 2013), the test set has both reverberation and noise. Therefore, the proposed methods in the challenge should perform both denoising and dereverberation. In Han et al. (2015), the authors proposed to perform dereverberation and denoising within a single DNN and substantially outperform related methods regarding quality and intelligibility. However, previous joint processing usually involved only two sub tasks. In our study, we joint perform four or more tasks to achieve general restoration.
Synthetic restoration Directly estimate the source signal from the observed mixture is hard especially when the SNR is low. Some studies adopted a regeneration approach. In Polyak et al. (2021), the authors utilized an ASR model, a pitch extraction model, and a loudness model to extract semantic level information from the speaker. Then these features were fed to an encode-decoder network to regenerate the speech signal. To maintain the consistency of speaker characteristics, it used an auxiliary identity network to compute the identity feature. Similar to synthetic speech restoration, TTS can be treated as the regeneration of speech from texts.

heading: A.1.2 NEURAL VOCODER

text: Vocoder, which maps the encoded speech features to the waveforms, is an indispensable component in speech synthesis. The most widely used input feature for vocoder is mel spectrogram. In recent years, since the emergence of WaveNet (Oord et al., 2016), neural network based vocoders demonstrate clear advantages over traditional parametric methods (Morise et al., 2016). Comparing with conventional methods, the quality of WaveNet is more closer to the human voice. Later, WaveRNN (Yu et al., 2019) is proposed to model the waveform with a single GRU. In this way, WaveRNN has much lower complexity comparing with WaveNet. However, the autoregressive nature of these models and deep structure make their inference process hard to be paralleled. To address this problem, non-autoregressive models like WaveGlow (Prenger et al., 2019) and WaveFlow (Ping et al., 2020) were proposed. Afterward, non-autoregressive GAN-based models such as MelGAN (Kumar et al., 2019) push the synthesis quality to a comparable level with autoregressive models. Recently, TFGAN (Tian et al., 2020) demonstrated strong capability in vocoding. Directed by multiple discriminators and loss functions, TFGAN was able to leverage information from both time domain and frequency domain. As a result, the synthesis quality of TFGAN is more natural and less metallic comparing with other GAN-based non-autoregressive models. In this work, we realize a universal vocoder based on TFGAN, which can reconstruct waveform from mel spectrogram of an arbitrary speaker with good perceptual quality. The pre-trained vocoder is available online to facilitate future studies and reproduce our work.

heading: B APPENDIX B

heading: B.1 DETAILS OF THE ANALYSIS STAGE

text:
The DNN and BiGRU we use are shown in Figure 7. DNN is a six layers fully connected network with BatchNorm and ReLU activations. The DNN accept each time step of the low-quality spectrogram as the input feature and output the restoration mask. Similarly, for the BiGRU model, we substitute some layers in DNN to a two-layer bidirectional GRU to capture the time dependency between time steps. To increase the modeling capacity of BiGRU, we expanded the input dimension of GRU to twice the mel frequency dimension with full connected networks.
The detailed architecture of ResUNet is shown in Figure 3a. In the down-path, the input low-quality mel spectrogram will go through 6 encoder blocks, which includes a stack of L1 ResConv and a 2 × 2 average pooling. In ResConv, the outputs of ConvBlock and the residual convolution are added together as the output. ConvBlock is a typical two layers convolution with BatchNorm and leakyReLU activation functions. The kernel size of residual convolution and the convolution in ConvBlock is 1 × 1 and 3 × 3. Correspondingly, the decoder blocks have the symmetric structure of the encoder blocks. It first performs a transpose convolution with 2× 2 stride and 3× 3 kernels, which result is concatenated with the output of the encoder at the same level to form the input of the decoder. The decoder also contain L2 layers of ResConv. The output of the final decoder block is passed to a final ConvBlock to fit the output channel.
We use Adam optimizer with β1 = 0.5, β2 = 0.999 and a 3e-4 learning rate to optimize the analysis stage of VoiceFixer. We treat the first 1000 steps as the warmup phase, during which the learning rate grows linearly from 0 to 3e-4. We decay the learning rate by 0.9 every 400 hours of training data. We perform an evaluation every 200 hours of training data. If we observe three consecutive evaluations with no improvement, we will interrupt the experiment.
For all the STFT and iSTFT, we use the hanning window with a window length of 2048 and a hop length of 441. As all the audio we use is at the 44.1 kHz sample rate, the corresponding spectrogram size in this setting will be T ×1025, where T is the dimension of time frames. For mel spectrogram, the dimensions of the linear spectrogram are transformed into T × 128.

heading: B.2 DETAILS OF THE SYNTHESIS STAGE

text:
As shown in Table 3, we use 7 kinds of STFT resolutions and 4 kinds of time resolution during the calculation of LF and LT . So KF = 7 in Equation 17 and KT = 4 in Equation 18.
The mel loss Lmel, spectral convergence loss Lsc, STFT magnitude loss Lmag, segment loss Lseg, energy loss Lenergy, and phase loss Lphase are defined in Equation 21 26. The function v(·) is the
k 1 2 3 4 5 6 7 win-length 4096 2048 1024 512 256 128 64 hop-length 2048 1024 512 256 128 64 32
fft-size 8192 4096 2048 1024 512 256 128
Table 2: STFT setup for different k in LF .
k 1 2 3 4 frame-length 1 240 480 960 hop-length 1 120 240 480
Table 3: Windowing setup for different k in LT .
windowing function that divide time sample into w windows and compute mean value within each window, v(s)1×w = (mean(s0),mean(s1), ...,mean(sw−1)). Each sw stand for windowed s. ∆ stand for first difference.
Lmel(ŝ, s) = ∥∥∥Ŝmel − Smel∥∥∥
2 (21)
Lsc(ŝ, s) = ∥∥∥|Ŝ| − |S|∥∥∥ F∥∥∥|Ŝ|∥∥∥
F
(22)
Lmag(ŝ, s) = ∥∥∥log(|Ŝ|)− log(|S|)∥∥∥
1 , (23)
Lseg(ŝ, s) = ‖v(ŝw)− v(sw)‖1 , (24) Lenergy(ŝ, s) = ∥∥v(ŝw2)− v(s2w)∥∥1 , (25)
Lphase(ŝ, s) = ∥∥∆v(ŝw2)−∆v(s2w)∥∥1 , (26)
Table 9 and Table 8 show the structure of frequency and time domain discriminators. The subband discriminators Dsub and multi-resolution time discriminators D (r) T (s) use the structure of Tdiscriminator, which is a stack of one dimensional convolution with grouping and large kernal size. The frequency discriminator DF use the similar module ResConv similar to ResUNet shown in Figure 3b.
F-discriminator Conv2d(1,32,kernal_size=(3,3))
ResConv(32, 32, stride=1,kernal_size=(3,3)) ResConv(32, 32, stride=1,kernal_size=(3,3)) ResConv(32, 64, stride=2,kernal_size=(3,3)) ResConv(64, 64, stride=1,kernal_size=(3,3)) ResConv(64, 32, stride=2,kernal_size=(3,3)) ResConv(32, 32, stride=1,kernal_size=(3,3)) ResConv(32, 32, stride=2,kernal_size=(3,3)) ResConv(32, 32, stride=1,kernal_size=(3,3))
Figure 9: The structure of F-discriminator.
For the training of vocoder, we setting up the λD to λseg value in Equation 16, Equation 17, and Equation 18 as λD = 4.0, λmel = 50, λsc = 5.0, λmag = 5.0, λenergy = 100.0, λphase = 100.0, and λseg = 200.0

heading: C APPENDIX C

heading: C.1 DATASETS

text:
Clean speech CSTR VCTK corpus (Yamagishi et al., 2019) is a multi-speaker English corpus containing 110 speakers with different accents. We split it into a training part VCTK-Train and a testing part VCTK-Test. The version of VCTK we used is 0.92. To follow the data preparation strategy of Lee & Han (2021), only the mic1 microphone data is used for experiments, and p280 and p315 are omitted for the technical issues. For the remaining 108 speakers, the last 8 speakers, p360,p361,p362,p363,p364,p374,p376,s5 are splitted as test set VCTK-Test. Within the other 100 speakers, p232 and p257 are omitted because they are used later in the test set DENOISE, the remaining 98 speakers are defined as VCTK-Train. Except for the training of NuWave, all the utterances are resampled at the 44.1 kHz sample rate. AISHELL-3 is an open-source Hi-Fi mandarin speech corpus, containing 88035 utterances with a total duration of 85 hours. HQ-TTS dataset contains 191 hours of clean speech data collected from a serial of datasets on openslr.org. In Table 4, we include the details of HQ-TTS, including the URL and language types of each subset.
Noise data One of the noise dataset we use come from VCTK-Demand (VD) (Valentini-Botinhao et al., 2017), a widely used corpus for speech denoising and noise-robust TTS training. This dataset contains a training part VD-Train and a testing part VD-Test, in which both contain two noisy set VD-Train-Noisy, VD-Test-Noisy and two clean speech set VD-Train-Clean, VD-Test-Clean. To obtain the noise data from this dataset, we minus each noisy data from VD-Train-Noisy with its corresponding clean part in VD-Train-Clean to get the final training noise dataset VD-Noise. The noise data are all resampled to 44.1 kHz. Another noise dataset we adopt is the TUT urban acoustic scenes 2018 dataset (Mesaros et al., 2018), which is originally used for the acoustic scene classification task of DCASE 2018 Challenge. The dataset contains 89 hours of high-quality recording from 10 acoustic scenes such as airport and shopping mall. The total amount of audio is divided into development DCASE-Dev and evaluation DCASE-Eval parts. Both of them contain audio from all cities and all acoustic scenes.
Room impulse response We randomly simulated a collection of Room Impulse Response filters to simulate the 44.1 kHz speech room reverberation using an open-source tool 2. The meters of height, width, and length of the room are sampled randomly in a uniform distribution U(1, 12). The placement of the microphone is then randomly selected within the room space. For the placement of the sound source, we first determined the distance to the microphone, which is randomly sampled in a Gaussian distribution N(µ, σ2), µ = 2, σ = 4. If the sampled value is negative or greater than five meters, we will sample the distance again until it meets the requirement. After sampling the distance between the microphone and sound source, the placement of the sound source is randomly selected within the sphere centered at the microphone. The RT60 value we choose come from the uniform distribution U(0.05, 1.0). For the pickup pattern of the microphone, we randomly choose from omnidirectional and cardioid types. Finally, we simulated 43239 filters, in which we randomly split out 5000 filters as the test set RIR-Test and named other 38239 filters as RIR-Train.
2https://github.com/sunits/rir_simulator_python

heading: C.2 TRAINING DATA

text:
We describe the simulation of training data in Algorithm 1. S = {s(0), s(1), ..., s(i)}, N = {n(0),n(1), ...,n(i)}, and R = {r(0), r(1), ..., r(i)} are the speech dataset, noise dataset, and RIR dataset. We use several helper function to describe this algorithm. randomFilterType(·) is a function that randomly select a type of filter within butterworth, chebyshev, bessel, and ellipic. Resample(x, o1, u) is a resampling function that resample the one dimensional signal x from a original samplerate o1 to the target u samplerate. buildFilter(t, c, o2) is a filter design function that return a type t filter with cutoff frequency c and order o2. max(·),min(·), and abs(·) is the element wise maximum, minimum, and absolute value function. mean(·) calculate the mean value of the input.
We first select a speech utterance s, a segment of noise n and a RIR filter r randomly from the dataset. Then with p1 probability, we add the reverberation effect using r. And with p2 probability, we add clipping effect with a clipping ratio η, which is sampled in a uniform distrubution U(ηlow, ηhigh). To produce low-resolution effect, after determining the filter type t, we randomly sample the cutoff frequency c and order o from the uniform distribution U(Clow, Chigh) and U(Olow, Ohigh). Then we perform convolution between x and the type t order o lowpass filter with cutoff frequency c. Finally the filtered data will be resampled twice, one is resample to c ∗ 2 samplerate and another is resample back to 44.1 kHz. We also perform the same lowpass filtering to the noise signal randomly. This operation is necessary because, if not, the model will overfit the pattern that the bandwidth of noise signal is always different from speech. In this case, the model will fail to remove noise when the bandwidth of noise and speech are similar. For the simulation of noisy environment, we randomly add the noise n into the speech signal x using a random SNR s ∼ U(Slow, Shigh). To fit the model with all energy levels, we randomly conduct a q ∼ U(Qlow, Qhigh) scaling to the input and target data pair. In our work, we choose the following parameters to perform this algorithm, p1 = 0.25, p2 = 0.25, p3 = 0.5, ηlow = 0.06, ηhigh = 0.9, Clow = 750, Chigh = 22050, Olow = 2, Ohigh = 10, Slow = −5, Shigh = 40, Qlow = 0.3, Qhigh = 1.0.
Algorithm 1: Add random distortions to high-quality speech signal s In: s← S; n← N; r ← R Out: The high-quality speech s and its randomly distorted version x
x = s; with p1 probability: x = x ∗ r ; /* Convolute with RIR filter */ with p2 probability:
θ ∼ U(Θlow,Θhigh) ; /* Choose clipping ratio */ x = max(min(x, θ),−θ) ; /* Hard clipping */
with p3 probability: t = randomFilterType() ; c ∼ U(Clow, Chigh) ; o ∼ U(Olow, Ohigh) ; /* Random cutoff and order */ x = x ∗ buildFilter(t, c, o) ; /* Low pass filtering */ x = Resample(Resample(x, 44100, c ∗ 2), c ∗ 2, 44100) ; /* Resample */ with p4 probability:
n = n ∗ buildFilter(t, c, o) ; /* Low pass filtering on noise */ n = Resample(Resample(n, c ∗ 2), 44100) ; /* Resample */
with p5 probability: s ∼ U(Slow, Shigh) ; q ∼ U(Qlow, Qhigh) ; /* Random SNR and scale */ n = nmean(abs(n))/mean(abs(x)) ; /* Normalize the energy of noise */ x = (x + n
10s/20 ) ; /* Add noise */
s = qs ; /* Scaling */ x = qx ; /* Scaling */

heading: C.3 TESTING DATA

text:
Testing data is crucial for the evaluation for each kind of distortion. The testing data we use either come from open-sourced test set or simulated by ourselves.
Super-resolution The simulation of the SR test set follows the work of (Kuleshov et al., 2017; Wang & Wang, 2021). The low-resolution and target data pairs are obtained by transforming 44.1 kHz sample rate utterances in target speech data VCTK-Test to a lower sample rate u. To achieve that, we first convolve the speech data with an order 8 Chebyshev type I lowpass filter with the u2 cutoff frequency. Then we subsample the signal to u sample rate using polyphase filtering. In this work, to test the performance on different sampling rate settings, u are set at 2 kHz, 4 kHz, 8 kHz, 16 kHz, and 24 kHz. We denote the corresponding five testing set as VCTK-4k, VCTK-4k, VCTK-8k, VCTK-16k, and VCTK-24k, respectively.
Denoising For the denoising task, we adopt the open-sourced testing set DENOISE described in Appendix C.1. This test set contains 824 utterances from a female speaker and a male speaker. The type of noise data comprises a domestic noise, an office noise, noise in the transport scene, and two street noises. The test set is simulated at four SNR levels, which are 17.5 dB, 12.5 dB, 7.5 dB, and 2.5 dB. The original data is sampled at 48 kHz. We downsample it to 44.1 kHz to fit our experiments.
Dereverberation The test set for dereverberation, DEREV, is simulated using VCTK-Test and RIRTest. For each utterance in VCTK-Test, we first randomly select an RIR from RIR-Test, then we calculate the convolution between the RIR and utterance to build the reverberant speech. Finally, we build 2937 reverberant and target data pairs.
Declipping DECLI, the evaluation set for declipping, is also constructed based on VCTK-Test. We perform clipping on VCTK-Test following the equation in Section 2 and choose 0.25, 0.1 as the two setups for the clipping ratio. This result in two declipping test sets with different levels, each containing 2937 clipped speech and target audios.
General speech restoration To evaluate the performance on GSR, we simulate a test set ALLGSR comprising of speech with all kinds of distortion. The clean speeches and noise data used to build ALL-GSR is VCTK-Test and DCASE-Eval. The simulation procedure of ALL-GSR is almost the same to the training data simulation described in Section 4.2. In total, 501 three seconds long utterances are produced in this test set.
MOS Evaluation We select a small portion from the test sets to carry out MOS evaluation for each one. In SR, DECLI, and DEREV, we select 38 utterances out for human ratings. In DENOISE and ALL-GSR, we randomly choose 42 and 51 utterances.

heading: C.4 EVALUATION METRICS

text:
Log-spectral distance LSD is a commonly used metrics on the evaluation of super-resolution performance (Kumar et al., 2020; Lee & Han, 2021; Wang & Wang, 2021). For target signal s and output estimate ŝ, LSD can be computed as Equation 27, where S and Ŝ stand for the magnitude spectrogram of s and ŝ.
LSD(S, Ŝ) = 1
T ΣTt=1
√ 1
F ΣFf=1 log10(
S(f, t)2 Ŝ(f, t)2 )2 (27)
Perceptual evaluation of speech quality PESQ is widely used in speech restoration literature as their evaluation metrics (Pascual et al., 2017; Hu et al., 2020). It was originally developed to model the subjective test commonly used in telecommunication. PESQ provides a score ranging from -0.5 to 4.5 and the higher the score, the better quality a speech has. In our work, we used an open-sourced implementation of PESQ to compute these metrics. Since PESQ only works on a 16 kHz sampling rate, we performed a 16 kHz downsampling to the output 44.1k audio before evaluation.
Structural similarity SSIM (Wang et al., 2004) is a metrics in image super-resolution. It addresses the shortcoming of pixel-level metrics by taking the image texture into account. We match the implementation of SSIM in (Wang et al., 2004) with ours and compute SSIM as Equation 28, where µS and σS is the mean and standard deviation of S. CovSŜ is the Covariance of S and Ŝ. 1 = 0.01
and 2 = 0.02 are two constant used to avoid zero division. Similarity is measured within the K 7*7 blocks divided from S and Ŝ.
SSIM(S, Ŝ) = ΣKk=1( (2µSkµŜk + 1)(2CovSkŜk + 2)
(µ2Sk + µ 2 Ŝk + 1)(σ2Sk + σ 2 Ŝk
+ 2) ) (28)
Scale-invariant signal to noise ratio SiSNR (Le Roux et al., 2019) is widely used in speech restoration literatures to compare the energy of a signal to its background noise. SiSNR is calculated between target waveform s and waveform estimation ŝ:
SiSNR(s, ŝ) = 10 ∗ log10 ‖ŝtarget‖2
‖enoise‖2 , (29)
where ŝtarget = <ŝ,s>s‖ŝ‖2 . A higher SiSNR indicates less discrepancy between the estimation and target.
Scale-invariant spectrogram to noise ratio SiSPNR is a spectral metric similar to SiSNR. They have the similar idea except SiSPNR is computed on the magnitude spectrogram. Given the target spectrogram S and estimation Ŝ, the computation of SiSPNR can be formulated as
SiSPNR(S, Ŝ) = 10 ∗ log10 ‖Ŝtarget‖2
‖Enoise‖2 (30)
where Ŝtarget = <Ŝ,S>S‖Ŝ‖2 . The scale invariant is guranteed by mean normalization of estimated and target spectrogram.

heading: D APPENDIX D

heading: D.1 EVALUATION RESULTS

heading: D.2 ANALYSIS STAGE PERFORMANCE

text:
In this section, we report the mel spectrogram restoration score on different test sets. They are used to evaluate the performane of the analysis stage. We calculate the LSD, SiSPNR, and SSIM values.
The Unprocessed column is calculated using the target and unprocessed mel spectrogram. And the Oracle-Mel column is calculated using the target spectrogram and itself.
Table 10 shows that on DENOISE, DEREV, and ALL-GSR, all four VoiceFixer based models are effective on the restoration of mel spectrogram. Among the four analysis stage models, UNet is consistently better than the other three models.
Table 11 lists the mel restoration performance on different sampling rates. Although VF-BiGRU has fewer parameters than VF-UNet, it still achieved the highest score on LSD and SiSPNR averagely. This result indicates that the recurrent structure might be more suitable for the mel spectrogram super-resolution task when the initial sampling rate is high.

heading: D.3 DEMOS

text:
In this section, we provide some restoration demos using our proposed VoiceFixer. In Figure 12, we provide eight restoration demos using our VF-UNet model. All the audios used in our demo are either collected from the internet or recorded by ourselves. In each example, the left is the unprocessed spectrogram, and the right is the restored one. After restoration, these seriously distorted speech signals can be revert to relatively high quality.
Figure 12b is the speech we recorded using Adobe Audition. We set the sampling rate to 8 kHz and manually add the clipping effect. It also contains some low-frequency noise and reverberation introduced by the recording device and environment. Figure 12a is a speech delivered by Amelia Earhart3, 1897-1937, appeared in the Library of Congress, United States. The original version sounds like a mumble. Figure 12f comes from an interview in a TV news program, which includes multiple distortions. Figure 12e is collected from the audio uploaded by a Youtuber4. Probably due to the recording device, her speech is deteriorated seriously by noise, and the energy of speech in the low-frequency part is also relatively low. Figure 12c is the restoration of a Chinese famous old movie railroad guerrilla5, which speech has limited bandwidth. The audio in Figure 12d is selected from a well-known TV series in China, romance of the three kindoms6, in which some parts of the spectrogram are masked off due to the previous audio compression. Figure 12g is a recording7 selected from a speech delivered by Sun Yat-sen, 1866-1925, which is in extremely lowresolution and includes multiple unknown distortions. Figure 12h shows the result of a subway broadcasting we recorded in Shanghai. The low-frequency part of speech almost lost completely, and the reverberation is serious.
To sum up, all these examples showcase the effectiveness of the VoiceFixer model on GSR. And to our surprise, it can generate will on unseen distortions such as the spectrogram lost in Figure 12c, Figure 12f, and Figure 12d. In addition, Figure 12e shows that VoiceFixer is effective for the compensation of low-frequency energy, making speech sound less machinery and distant. Last but not least, despite the abnormal harmonic structure in the low-frequency part in Figure 12g, our proposed model can still restore its sound, which display the advantages of utilizing the prior knowledge of vocoder.
3https://www.loc.gov/item/afccal000004 4https://v.ixigua.com/egVW74E/ 5https://www.youtube.com/watch?v=R8lY1qn2CHA 6https://www.youtube.com/watch?v=6h7N4Cl1lTw 7https://www.bilibili.com/video/BV1WW411V7fR

title: VOICEFIXER: TOWARD GENERAL SPEECH RESTORA-