We are interested in a novel task, singing voice beautification (SVB). Given the singing voice of an amateur singer, SVB aims to improve the intonation and vocal tone of the voice, while keeping the content and vocal timbre. Current automatic pitch correction techniques are immature, and most of them are restricted to intonation but ignore the overall aesthetic quality. Hence, we introduce Neural Singing Voice Beautifier (NSVB), the first generative model to solve the SVB task, which adopts a conditional variational autoencoder as the backbone and learns the latent representations of vocal tone. In NSVB, we propose a novel time-warping approach for pitch correction: Shape-Aware Dynamic Time Warping (SADTW), which ameliorates the robustness of existing time-warping approaches, to synchronize the amateur recording with the template pitch curve. Furthermore, we propose a latent-mapping algorithm in the latent space to convert the amateur vocal tone to the professional one. To achieve this, we also propose a new dataset containing parallel singing recordings of both amateur and professional versions. Extensive experiments on both Chinese and English songs demonstrate the effectiveness of our methods in terms of both objective and subjective metrics. Audio samples are available at https://neuralsvb. github.io . Codes: https://github. com/MoonInTheRiver/NeuralSVB . The major successes of the artificial intelligent singing voice research are primarily in Singing Voice Synthesis (SVS) (Lee et al., 2019; Blaauw and Bonada, 2020; Ren et al., 2020; Lu et al., 2020; Liu et al., 2021a) and Singing Voice Conversion (SVC) (Sisman and Li, 2020; Li et al., 2021; Wang et al., 2021a). However, the Singing Voice Beautification (SVB) remains an important and challenging endeavor for researchers. SVB aims to improve the  Corresponding Author intonation 1 and the vocal tone of the voice, while keeping the content and vocal timbre 2 . SVB is extensively required both in the professional recording studios and the entertainment industries in our daily life, since it is impractical to record flawless singing audio. Nowadays in real-life scenarios, SVB is usually performed by professional sound engineers with adequate domain knowledge, who manipulate commercial vocal correction tools such as Melodyne 3 and Autotune 4 (Yong and Nam, 2018). Most current automatic pitch correction works are shown to be an attractive alternative, but they may 1) show weak alignment accuracy (Luo et al., 2018) or pitch accuracy (Wager et al., 2020); 2) cause the tuned recording and the reference recording to be homogeneous in singing style (Yong and Nam, 2018). Besides, they typically focus on the intonation but ignore the overall aesthetic quality (audio quality and vocal tone) (Rosenzweig et al., 2021; Zhuang et al., 2021). To tackle these challenges, we introduce Neural Singing Voice Beautifier (NSVB), the first generative model to solve the SVB task, which adopts a Conditional Variational AutoEncoder (CVAE) (Kingma and Welling, 2014; Sohn et al., 2015) as the backbone to generate high-quality audio and learns the latent representation of vocal tone. In NSVB, we dichotomize the SVB task into pitch correction and vocal tone improvement: 1) To correct the intonation, a straightforward method is aligning the amateur recording with the template pitch curve, and then putting them together to resynthesize a new singing sample. Previous 1 Intonation refers to the accuracy of pitch in singing. 2 The differences between the vocal tone and vocal timbre is that: the former represents one's skills of singing, such as airflow controlling ability, muscle strength of vocal folds and vocal placement; the latter represents the identical, overall sound of one's vocal. 3 https://www.celemony.com/en/start 4 https://www.antarestech.com/ 7970 works (Wada et al., 2017; Luo et al., 2018) implemented this by figuring out the alignment through Dynamic Time Warping (DTW) (Mller, 2007) or Canonical Time Warping (CTW) (Zhou and Torre, 2009). We propose a novel Shape-Aware DTW algorithm, which ameliorates the robustness of existing time-warping approaches by considering the shape of the pitch curve rather than low-level features when calculating the optimal alignment path. 2) To improve the vocal tone, we propose a latent-mapping algorithm in the latent space, which converts the latent variables of the amateur vocal tone to those of the professional ones. This process is optimized by maximizing the log-likelihood of the converted latent variables. To retain the vocal timbre during the vocal tone mapping, we also propose a new dataset named PopBuTFy containing parallel singing recordings of both amateur and professional versions. Besides, thanks to the autoencoder structure, NSVB inherently supports semi-supervised learning, where the additional unpaired, unlabeled 5 singing data could be leveraged to facilitate the learning of the latent representations. Extensive experiments on both Chinese and English songs show that NSVB outperforms previous methods by a notable margin, and each component in NSVB is effective, in terms of both objective and subjective metrics. The main contributions of this work are summarized as follows:  We propose the first generative model NSVB to solve the SVB task. NSVB not only corrects the pitch of amateur recordings, but also generates the audio with high audio quality and improved vocal tone, to which previous works typically pay little attention. We propose Shape-Aware Dynamic Time Warping (SADTW) algorithm to synchronize the amateur recording with the template pitch curve, which ameliorates the robustness of the previous time-warping algorithm. We propose a latent-mapping algorithm to convert the latent variable of the amateur vocal tone to the professional one's, and contribute a new dataset PopBuTFyto train the latent-mapping function. 5 unpaired, unlabeled means the recordings sung by any people, in any vocal tone without label. Singing Voice Conversion (SVC) is a sub-task of Voice Conversion (VC) (Berg-Kirkpatrick and Klein, 2015; Serr et al., 2019; Popov et al., 2021; Liu et al., 2021b), which transforms the vocal timbre (or singer identity) of one singer to that of another singer, while preserving the linguistic content and pitch/melody information (Li et al., 2021). Mainstream SVC models can be grouped into three categories (Zhao et al., 2020): 1) parallel spectral feature mapping models, which learn the conversion function between source and target singers relying on parallel singing data (Villavicencio and Bonada, 2010; Kobayashi et al., 2015; Sisman et al., 2019); 2) Cycle-consistent Generative Adversarial Networks (CycleGAN) (Zhu et al., 2017; Kaneko et al., 2019), where an adversarial loss and a cycle-consistency loss are concurrently used to learn the forward and inverse mappings simultaneously (Sis-man and Li, 2020); 3) encoder-decoder models, such as PPG-SVC (Li et al., 2021), which leverage a singing voice synthesis (SVS) system for SVC (Zhang et al., 2020), and auto-encoder (Qian et al., 2019a; Wang et al., 2021b; Yuan et al., 2020) based SVC (Wang et al., 2021a). The models of the latter two categories can be utilized with nonparallel data. In our work, we aim to convert the intonation and the vocal tone while keeping the content and the vocal timbre, which is quite different from the SVC task. Automatic Pitch Correction (APC) works attempt to minimize the manual effort in modifying the flawed singing voice (Yong and Nam, 2018). Luo et al. (2018) propose Canonical Time Warping (CTW) (Zhou and Torre, 2009; Zhou and De la Torre, 2012) which aligns amateur singing recordings to professional ones according to the pitch curves only. Wager et al. (2020) propose a data-driven approach to predict pitch shifts depending on both amateur recording and its accompaniment. Rosenzweig et al. (2021) propose a pitch shift method for Cappella recordings. Zhuang et al. (2021) propose a pitch-controllable SVS system to resynthesize the audio with correctly predicted pitch curves. Besides modifying pitch, Yong and 7971 Stage 1 Pitch (Aligned in Stage 2 and Inference) VAE Dec VAE Enc Discriminator Pitch Module Z Type equation here. The training process consists of 2 stages, and the second stage shares the same pipeline with the inference stage. VAE Enc means the encoder of CVAE; VAE Dec means the decoder of CVAE; Mel means the mel-spectrogram;  z  means the latent variable of the vocal tone; the  a / p  subscript means the amateur/professional version. Nam (2018) propose to modify pitch and energy information to improve the singing expressions of an amateur singing recording. However, this method heavily relies on a reference recording, causing the tuned recording and the reference recording to be homogeneous in singing style (Zhuang et al., 2021). Our work adopts the non-parametric and data-free pitch correction method like Luo et al. (2018), but improves the accuracy of alignment. In this section, we describe the overview of NSVB, which is shown in Figure 1. At Stage 1 in the figure, we reconstruct the input mel-spectrogram through the CVAE backbone (Section 3.1) based on the pitch, content and vocal timbre conditions extracted from the input by the pitch encoder, content encoder and timbre encoder, and optimize the CVAE by maximizing evidence lower bound and adversarial learning. At Stage 2/Inference in the figure, firstly we infer the latent variable z a based on the amateur conditions; secondly we prepare the amateur content vectors aligned with the professional pitch by SADTW algorithm (Section 3.2); thirdly we map z a to z p by the latent-mapping algorithm (Section 3.3); finally, we mix the professional pitch, the aligned amateur content vectors, and the amateur vocal timbre to obtain a new condition, which is leveraged along with the mapped z p by the decoder of CVAE to generate a new beautified mel-spectrogram. The training/inference details and model structure of each component in NSVB are described in Section 3.4 and Section 3.5. As shown in Figure 2, to generate audio with high quality and learn the latent representations of vocal tone, we introduce a Conditional Variational AutoEncoder (CVAE) (Kingma and Welling, 2014; Sohn et al., 2015) as the mel-spectrogram generator, with the optimizing objective of maximizing the evidence lower bound (ELBO) of the intractable marginal log-likelihood of mel-spectrogram log p  ( x | c ) : log p  ( x | c )  ELBO ( ,  )  E z  q  ( z | x ,c ) (cid:20) log p  ( x | z , c )  log q  ( z | x , c p ( z ) where x , c , z denote the input/output mel-spectrogram, the mix of content, vocal timbre and pitch conditions, and the latent variable representing the vocal tone respectively;  and  denote the model parameters of CVAE encoder and CVAE decoder; q  ( z | x , c ) is the posterior distribution approximated by the CVAE encoder; p  ( x | z , c ) is the likelihood function that generates mel-spectrograms given latent variable z and condition c ; p ( z ) is the prior distribution of the latent variables z , and we choose the standard normal distribution as p ( z ) for simplification. Furthermore, to address the over-smoothing problem (Qian et al., 2019b) in CVAE, we utilize an adversarial discriminator ( D ) (Mao et al., 2017) to refine the output mel-spectrogram: L adv ( ,  ) = E [( D ( (cid:101) x )  1) 2 ] , L adv ( D ) = E [( D ( x )  1) 2 ] + E [ D ( (cid:101) x ) 2 ] , (1) 7972 where x is the ground-truth and (cid:101) x is the output of CVAE. The descriptions for the model structure of each component are in Section 3.5. To implement the pitch correction, a straightforward method is aligning the amateur recording with the template pitch curve, and then concatenating them to resynthesize a new singing sample with improved intonation. Since the source pitch curve of amateur recordings and template one show a high degree of natural correlation along the time axis, applying a proper time-warping algorithm on them is crucial. However, original DTW (Mller, 2007) could result in a poor alignment when certain parts of the axis move to higher frequencies, and other parts to lower ones, or vice versa (Sunder-mann and Ney, 2003). Luo et al. (2018) adopt an advanced algorithm CTW (Zhou and Torre, 2009), which combines the canonical correlation analysis (CCA) and DTW to extract the feature sequences of two pitch curves, and then apply DTW on them. However, the alignment accuracy of CTW leaves much to be desired. We elaborate a non-parametric and data-free algorithm, Shape-Aware DTW (SADTW), based on the prior knowledge that the source pitch curve and the template one have analogous local shape contours. Specifically, we replace the Euclidean distance in the original DTW distance matrix with the shape context descriptor distance. The shape context descriptor of a time point f i in one pitch curve is illustrated in Figure 3. Inspired by (Mori et al., 2005), we divide the data points around f i into m  n bins by m time windows and n angles. We calculate the number of all points falling in the k -th bin. Then the descriptor for f i is defined as the histogram h i  R m  n : h i ( k ) = |{ f j  = f i , f j  bin ( k ) }| , where |  | means the cardinality of a set. This histogram represents the distribution over relative positions, which is a robust, compact and discriminative descriptor. Then, it is natural to use the X 2 -test statistic on this distribution descriptor as the distance of two points f a and f p : C ( a, p ) = 1 2 m  n (cid:88) k =1 [ h a ( k )  h p ( k )] 2 h a ( k ) + h p ( k ) , where h a and h p are the normalized histograms corresponding to the point f a from the amateur pitch curve and the point f p from the template pitch curve. C ( a, p ) ranges from 0 to 1. Finally, we run DTW on the distance matrix C to obtain the alignment with least distance cost between two curves. Define a pair of mel-spectrograms ( x a , x p ) : the contents of x a and y p are the same sentence of a song from the same singer 6 , who sings these two recordings using the amateur tone and the professional tone respectively. Given the CVAE model, 6 The singers all major in vocal music. we can infer the posterior distribution q  ( z a | x a , c a ) and q  ( z p | x p , c p ) corresponding to x a and x p through the encoder of CVAE. To achieve the conversion of vocal tone, we introduce a mapping function M to convert the latent variables from q  ( z a | x a , c a ) to q  ( z p | x p , c p ) . Concretely, we sample a latent variable of amateur vocal tone z a from q  ( z a | x a , c a ) , and map z a to M ( z a ) . Then, M can be optimized by minimizing the negative log-likelihood of M ( z a ) : L map 1 ( M ) =  log q  ( M ( z a ) | x p , c p ) . Define  c p as the mix of 1) the content vectors from the amateur recording aligned by SADTW, 2) vocal timbre embedding encoded by timbre encoder, and 3) template pitch 7 embeddings encoded by pitch encoder. To make sure the converted latent variable could work well together with  c p to generate a high-quality audio sample (with the correct pitch and improved vocal tone), we send M ( z a ) to the CVAE decoder to generate  x , and propose an additional loss: L map 2 ( M ) =   x  x p  1 +  ( D ( x )  1) 2 , where D has been optimized by Eq. There are two training stages for NSVB: in the first training stage, we optimize CVAE by minimizing the following loss function and optimize the discriminator ( D ) by minimizing Eq. (1). Note that, the first stage is the reconstruction process of mel-spectrograms, where any unpaired, unlabeled singing data beyond PopBuTFy could be leveraged to facilitate the learning of the latent representations. In the second training stage, we optimize M on the parallel dataset PopBuTFy by minimizing the following loss function L ( M ) = L map 1 ( M ) + L map 2 ( M ) . ,  , and D are not optimized in this stage. In inference, the encoder of CVAE encodes x a with the condition c a to predict z a . Secondly, we map z a to M ( z a ) , and run SADTW to align the 7 During training, template pitch is extracted from the waveform corresponding to x p . amateur recordings with the template pitch curve. The template pitch curve can be derived from a reference recording with good intonation or a pitch predictor with the input of music notes. Then, we obtain  c p defined in Section 3.3 and send M ( z a ) together with  c p in the decoder of CVAE to generate  x . Finally, by running a pre-trained vocoder conditioned on  x , a new beautified recording is produced. 3.5 Model Structure The encoder of CVAE consists of a 1-D convolutional layer (stride=4), an 8-layer WaveNet structure (Oord et al., 2016; Rethage et al., 2018) and 3 1-D convolutional layers (stride=2) with ReLU activation function and batch normalization followed by a mean pooling, which outputs the mean and log scale standard deviation parameters in the posterior distribution of z . The decoder of CVAE consists of a 4-layer WaveNet structure and a 1-D convolutional layer, which outputs the mel-spectrogram with 80 channels. The discriminator adopts the same structure as (Wu and Luan, 2020), which consists of multiple random window discriminators. The latent-mapping function is composed of 2 linear layers to encode the vocal timbre as the mapping condition, and 3 linear layers to map z a . The pitch encoder is composed of 3 convolutional layers. In addition, given a singing recording, 1) to obtain its content vectors, we train an Automatic Speech Recognition (ASR) model based on Conformer (Gulati et al., 2020) with both speech and singing data, and extract the hidden states from the ASR encoder (viewed as the content encoder) output as the linguistic content information, which are also called phonetic posterior-grams (PPG); 2) to obtain the vocal timbre, we leverage the open-source API resemblyzer 8 as the timbre encoder, which is a deep learning model designed for speaker verification (Wan et al., 2018), to extract the identity information of a singer. More details of model structure can be found in Appendix A. 4 Experiments 4.1 Experimental Setup In this section, we first introduce PopBuTFy, the dataset for SVB, and then describe the implementation details in our work. Finally, we explain the evaluation method we adopt in this paper. Dataset Since there is no publicly available high-quality, unaccompanied and parallel singing dataset for the SVB task, we collect and annotate a dataset containing both Chinese Mandarin and English pop songs: PopBuTFy. To collect PopBuTFy for SVB, the qualified singers majoring in vocal music are asked to sing a song twice, using the amateur vocal tone for one time and the professional vocal tone for another. Note that some of the amateur recordings are sung off-key by one or more semi-tones for the pitch correction sub-task. The parallel setting could make sure that the personal vocal timbre will keep still during the beautification process. In all, PopBuTFy consists of 99 Chinese pop songs (  10.4 hours in total) from 12 singers and 443 English pop songs (  40.4 hours in total) from 22 singers. All the audio files are recorded in a professional recording studio by qualified singers, male and female. Every song is sampled at 22050 Hz with 16-bit quantization. We randomly choose 6 songs in Chinese and 18 songs in English (from unseen speakers) for validation and test. For subjective evaluations, we choose 60 samples in the test set from different singers, half in Chinese and English. All testing samples are included for objective evaluations. Implementation Details We train the Neural Singing Beautifier on a single 32G Nividia V100 GPU with the batch size of 64 sentences for both 100k steps in Stage 1 and Stage 2 respectively. Besides PopBuTFy, we pre-train the ASR model (used for PPG extraction) leveraging the extra speech datasets: AISHELL-3 (Yao Shi et al., 2020) for Chinese and LibriTTS (Zen et al., 2019) for English. For the semi-supervised learning mentioned in Section 1 and Section 3.4, we leverage an internal Chinese singing dataset (  30 hours without labeled vocal tone) in the first training stage described in Section 3.4 for Chinese experiments. The output mel-spectrograms of our model are transformed into audio samples using a HiFi-GAN vocoder (Kong et al., 2020) trained with singing data in advance. We set the  metioned in Section 3.3 to 0 . 1 . We transform the raw waveform with the sampling rate 22050 Hz into mel-spectrograms with the frame size 1024 and the hop size 128. We extract F 0 (fundamental frequency) as pitch information from the raw waveform using Parselmouth 9 , following Wu and Luan (2020); Blaauw and Bonada (2020); 9 https://github.com/YannickJadoul/ Parselmouth Ren et al. (2020). To obtain the ground truth pitch alignment between the amateur recordings and the professional ones for evaluating the accuracy of pitch alignment algorithm, we run the Montreal Forced Aligner tool (McAuliffe et al., 2017) on all the singing recordings to obtain their alignments to lyrics. Then the ground-truth pitch alignment can be derived since the lyrics are shared in a pair of data in PopBuTFy. Performance Evaluation We employ both subjective metrics: Mean Opinion Score (MOS), Comparison Mean Opinion Score (CMOS), and an objective metric: Mean Cepstral Distortion (MCD) to evaluate the audio quality on the test-set. Besides, we use F0 Root Mean Square Error (F0 RMSE) and Pitch Alignment Accuracy (PAA) to estimate the pitch correction performance. For audio, we analyze the MOS and CMOS in two aspects: audio quality (naturalness, pronunciation and sound quality) and vocal tone quality. MOS-Q/CMOS-Q and MOS-V/CMOS-V correspond to the MOS/CMOS of audio quality and vocal tone quality respectively. More details about subjective evaluations are placed in Appendix C. 4.2 Main Results In this section, we conduct extensive experiments to present our proposed model in regard to 1) the performance of pitch conversion; 2) the audio quality and vocal tone quality. Firstly, we provide the comparison among time-warping algorithms in terms of PAA in Table 1. Normed DTW means two pitch curves will be normalized before running DTW (Mller, 2007); CTW means the Canonical Time Warping (Zhou and Torre, 2009), which is used for pitch correction in Luo et al. (2018). It can be seen that, SADTW surpasses existing methods by a large margin. We also visualize an alignment example of DTW , CTW , and SADTW in Figure 4. Secondly, to check whether the amateur recordings are corrected to the good intonation after being beautified by NSVB, we calculate the F0 RMSE metric of the amateur recordings and the audio generated by NSVB, and list the results in Table 2. We can see that F0 RMSE has been improved significantly, which means NSVB successfully achieve pitch correction. To thoroughly evaluate our proposed model in audio quality and vocal tone quality, we compare subjective metric MOS-Q, MOS-V and objective metric MCD of audio samples generated by NVSB with the systems including: 1) GT Mel , amateur (A) and professional (P) version, where we first convert ground truth audio into mel-spectrograms, and then convert the mel-spectrograms back to audio using HiFi-GAN introduced in Section 4.1; 2) Baseline : the baseline model for SVB based on WaveNet with the number of parameters similar to NSVB , which adopts the same pitch correction method (SADTW) as NSVB does, and takes in the condition  c p defined in Section 3.3 to generate the mel-spectrogram optimized by the L 1 distance to x p . MCD is calculated using the audio samples of GT Mel P as references. The subjective and objective results on both Chinese and English datasets are shown in Table 3. We can see that 1) NSVB achieves promising results, with MOS-Q being less than those for ground truth professional recordings by only 0.1 and 0.12 on both datasets; 2) NSVB surpasses the GT Mel A in terms of MOS-V by a large margin, which indicates that NSVB successfully accomplishes the vocal tone improvement. 3) NSVB surpasses the baseline model on all the metrics distinctly, which proves the superiority of our proposed model; 4) GT Mel P , NSVB and Baseline all outperform GT Mel A in terms of MOS-V, which demonstrates that the proposed dataset PopBuTFy is reasonably labeled in respect of vocal tone. We conduct some ablation studies to demonstrate the effectiveness of our proposed methods and some designs in our model, including latent-mapping, additional loss L map 2 in the second training stage, and semi-supervised learning with extra unpaired, unlabeled data on Chinese songs. We compare audio samples from NSVB with and without latent-mapping in terms of CMOS-V and MCD. From Table 4, we can see that the latent-mapping brings CMOS-V and MCD gains, which demonstrates the improvements in vocal tone from latent-mapping in our model. We visualize linear-spectrograms of GT Mel A , GT Mel P , NSVB , NSVB w/o mapping in Appendix B. The patterns of high-frequency parts in NVSB samples are comparatively similar to those in GT Mel P samples while NSVB w/o mapping sample resembles GT Mel A samples. As shown in Table 5, all the compared metrics show the effectiveness of L map 2 , which means that the additional loss L map 2 is beneficial to optimizing the latent mapping function M , working as a complement to the basic loss L map 1 . To illustrate the advantage of the CVAE architecture that allows semi-supervised training, we compare NSVB trained with and without extra unpaired, unlabeled data on Chinese songs. The corresponding results are shown in Table 6. The compared metrics indicate the advantage of semi-supervised learning, which facilitates the learning of the latent representations for better sample reconstruction (audio quality) and better latent conversion (vocal tone quality). In this work, we propose Neural Singing Voice Beautifier, the first generative model for the SVB task, which is based on a CVAE model allowing semi-supervised learning. For pitch correction, we propose a robust alignment algorithm: Shape-Aware Dynamic Time Warping (SADTW). For vocal tone improvement, we propose a latent mapping algorithm. To retain the vocal timbre during the vocal tone mapping, we also propose a new specialized SVB dataset named PopBuTFy containing parallel singing recordings of both amateur and professional versions. The experiments conducted on the dataset of Chinese and English songs show that NSVB accomplishes the SVB task (pitch correction and vocal tone improvement), and extensional ablation studies demonstrate the effectiveness of the proposed methods mentioned above. This work was supported in part by Zhejiang Natural Science Foundation under Grant LR19F020006, National Natural Science Foundation of China under Grant No.61836002, No.62072397. Thank all the co-authors for their wonderful contributions, the enlightening opinions of the participants in the discussion, and the great efforts of singers and data annotators.
objective abstain abstain objective objective objective objective objective other abstain other abstain other abstain abstain abstain abstain abstain abstain abstain objective abstain abstain other objective objective abstain objective abstain abstain objective objective objective objective abstain other other other objective other other other other other other other other other other abstain abstain method other abstain other abstain method method abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain other abstain abstain abstain abstain abstain abstain method abstain abstain other abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain other abstain abstain abstain abstain other abstain method method abstain abstain abstain abstain abstain abstain method method abstain method method abstain method abstain abstain method other abstain abstain abstain method method abstain abstain method abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain objective objective objective objective abstain other other