Zhou Zhao  Zhejiang University zhaozhou@zju.edu.cn Abstract Non-autoregressive text to speech (NAR-TTS) models have attracted much attention from both academia and industry due to their fast generation speed. One limitation of NAR-TTS models is that they ignore the correlation in time and frequency domains while generating speech mel-spectrograms, and thus cause blurry and over-smoothed results. In this work, we revisit this over-smoothing problem from a novel perspective: the degree of over-smoothness is determined by the gap between the complexity of data distributions and the capability of modeling methods. Both simplifying data distributions and improving modeling methods can alleviate the problem. Accordingly, we first study methods reducing the complexity of data distributions. Then we conduct a comprehensive study on NAR-TTS models that use some advanced modeling methods. Based on these studies, we find that 1) methods that provide additional condition inputs reduce the complexity of data distributions to model, thus alleviating the over-smoothing problem and achieving better voice quality. 2) Among advanced modeling methods, Laplacian mixture loss performs well at modeling multimodal distributions and enjoys its simplicity, while GAN and Glow achieve the best voice quality while suffering from increased training or model complexity. 3) The two categories of methods can be combined to further alleviate the over-smoothness and improve the voice quality. 4) Our experiments on the multi-speaker dataset lead to similar conclusions as above and providing more variance information can reduce the difficulty of modeling the target data distribution and alleviate the requirements for model capacity. Non-autoregressive text to speech (NAR-TTS) models (Ren et al., 2019, 2020; Peng et al., 2020; Vainer and Duek, 2020; ancucki, 2020; Kim et al., 2020; Miao et al., 2020) have shown much faster inference speed than their autoregressive counterparts (Wang et al., 2017; Shen et al., 2018; Ping et al., 2018), while achieving comparable or even better voice quality (Ren et al., 2019, 2020). The text-to-speech mapping can be formulated as a conditional distribution P ( y | x ) where x and y are the text and speech sequences, respectively. Text-to-speech mapping is a one-to-many mapping problem (Wang et al., 2017), since multiple possible speech sequences correspond to a text sequence due to speech variations such as pitch, duration and prosody. Furthermore, speech mel-spectrograms are strongly correlated in time and frequency dimensions (see Section 2.1 for detailed analyses). Therefore, P ( y | x ) is actually a dependent and multimodal distribution (Ling et al., 2013; Zen and Senior, 2014) 1 . Early non-autoregressive TTS models (Ren et al., 2019; Peng et al., 2020) use mean absolute error (MAE) or mean square error (MSE) as loss function to model speech mel-spectrograms, implicitly assuming that data points in mel-spectrograms are independent to each other and follow a unimodal distribution 2 . Consequently, the mel-spectrograms following dependent and multimodal distributions cannot be well modeled by the MAE or MSE loss, which presents great challenges in non-autoregressive TTS modeling and causes over-smoothed (blurred) predictions in mel-spectrograms (Vasquez and Lewis, 2019; Sheng and Pavlovskiy, 2019). In this work, we conduct a comprehensive study on the over-smoothing problem in TTS. We find that the over-smoothness is closely related to the mismatch between the complexity of data distri-1 Here "dependent" means that the different dimensions (in either temporal domain or frequency domain) of y are dependent to each other. 2 MAE can be derived from the Laplace distribution and MSE from the Gaussian distribution, both of which are unimodal. butions ( e . g ., dependent and multimodal distributions are more complex than independent and unimodal distributions) and the power of modeling methods ( e . g ., simple losses such as MAE and MSE are less powerful than GAN or Glow-based methods). Both simplifying data distributions and enhancing modeling methods can alleviate the over-smoothing problem. From this perspective, we categorize recent methods combating over-smoothness into two classes: 1. Simplifying data distributions : The data distribution P ( y | x ) can be simplified by providing more conditional input information. We review two main methods: 1) Providing the previous mel-spectrogram frames y <t to predict current frame y t , i.e., factorizing the complex dependent distribution P ( y | x ) into a simpler conditional distribution (cid:81) t P ( y t | y <t , x ) , as used in autoregressive TTS models (Wang et al., 2017; Li et al., 2019). 2) Providing more variance information v 3 such as pitch, duration, and energy to predict mel-spectrogram in parallel, i.e., modeling P ( y | x, v ) rather than P ( y | x ) , as done in some non-autoregressive TTS models (Ren et al., 2020; ancucki, 2020). 2. Enhancing modeling methods : Generally speaking, the modeling method should be powerful enough to fit complex data distributions. We review methods based on different distribution assumptions, including Laplacian mixture loss, structural similarity index (SSIM) (Wang et al., 2004) loss, generative adversarial network (GAN) (Lee et al., 2020) and Glow (Kim et al., 2020). By studying those methods, we have the following findings. We hope that our studies and findings can inspire the community to design better models for TTS. By either autoregressive factorization or providing more variance information as input, complex distributions can be simplified to be less dependent and multimodal, which clearly alleviates the over-smoothing problem and improves the generated voice quality. Among them, providing more variance information such as FastSpeech 2 enjoys the advantages of fast generation due to its non-autoregressive nature. 3 The term "variance information" is first mentioned in FastSpeech 2 (Ren et al., 2020), which refers some speechrelated conditional information loss significantly improves the quality of generated mel-spectrograms and enjoys the simplicity of modeling. GAN and Glow achieve the best quality under both subjective and objective evaluations (since they make no assumptions about output distributions), but at the cost of increased training or model complexity. To further analyze the effectiveness of combining the basic ideas of the above two categories, we enhance FastSpeech 2 (considering its fast inference speed and good quality in the first category) with Laplacian mixture loss, SSIM, GAN and Glow, respectively. We find that the enhanced FastSpeech 2 generates speech with even better quality and the over-smoothing problem is further alleviated, which shows that the methods in the two categories are complementary to each other. We also extend our experiments to multi-speaker TTS task and obtain similar conclusions as above. Besides, we find that Glow has poor modeling ability in multi-speaker scenarios due to limited model capacity and more complex target data distributions compared with the single-speaker scenario, while it can be greatly alleviated by simplifying data distributions (introducing more variance information). Text-to-speech mapping is a one-to-many mapping since multiple speech sequences can possibly correspond to a text sequence with different pitch, duration and prosody, making the mel-spectrograms distribution P ( y | x ) multimodal. And due to the continuous nature of the mel-spectrograms, adjacent data points are dependent to each other. In this section, we first empirically characterize the distribution of P ( y | x ) in TTS 4 through visualization (Section 2.1), and then provide a novel perspective to study the over-smoothing problem in TTS (Section 2.2). We first visualize the distribution of P ( y | x ) to see whether it is dependent and multimodal 5 . Denote 4 In this work, we mainly focus on text (phoneme) to mel-spectrogram mapping, and leave mel-spectrogram to waveform mapping and text to waveform mapping to future work. We approximate the mel-spectrogram distribution on LJSpeech dataset. The detailed data processing procedure is the same as that in Section 3.2. the data point in the t -th frame and the f -th frequency bin in the ground-truth mel-spectrogram as y ( t, f ) , where t  [1 , T ] , f  [1 , F ] , and T , F represent the total length and the number of frequency bins of mel-spectrograms respectively. Since different phonemes have different mel-spectrograms, we analyze the distribution of each phoneme separately. Specifically, for all the mel-spectrogram frames corresponding to each phoneme ph , we calculate three distributions: 1) marginal distribution P ( y ( t, f ) | x = ph ) ; 2) joint distribution between two different frequency bins P ( y ( t, f 1 ) , y ( t, f 2 ) | x = ph ) ; 3) joint distribution between two different time frames P ( y ( t 1 , f ) , y ( t 2 , f ) | x = ph ) 6 . For each distribution, we first compute the histograms and smooth into probability density functions with kernel density estimation (Dehnad, 1987) for better visualization. The marginal distributions P ( y ( t, f ) | x = ph ) for several different phonemes are shown in Figure 1. It can be seen that the shape of the marginal distribution of each data point y ( t, f ) in mel-sepctrogram is multimodal, especially for data points in high-frequency bins. The joint distribution P ( y ( t, f 1 ) , y ( t, f 2 ) | x = ph ) and P ( y ( t 1 , f ) , y ( t 2 , f ) | x = ph ) are shown in Figure 2a and Figure 2b, respectively. Obviously, those 6 We denote one of the frame index among all mel-spectrograms corresponding to the phoneme x as t 1 and set t 2 = t 1 + 1 . joint distributions are also multimodal and neighboring points in mel-spectrograms are strongly correlated. From these observations, we can see that the distribution P ( y | x ) of mel-sepctrograms is multimodal and dependent across time and frequency. The dependent and multimodal distribution of P ( y | x ) increases the difficulty of TTS modeling and causes over-smoothing problem if it is not correctly handled. We provide a novel perspective to depict this problem: the degree of over-smoothness is closely related to the gap between the complexity of data distributions and the power of modeling methods. A larger gap between the power of a modeling method and the complexity of a data distribution results in more severe over-smoothing problem. Consequently, simplifying data distributions and enhancing modeling methods can alleviate the over-smoothing problem. From this perspective, 8199 we list the methods to combat over-smoothness in two categories in Table 1. In the following sections, we first explore the effectiveness of simplifying data distributions (Sec-tion 3) and then that of enhancing modeling methods (Section 4). Finally, we combine the basic ideas of these two categories to improve an existing model (Section 5.1) and conduct further exploration on the multi-speaker dataset (Section 5.2). | Simplifying data distributions P ( y | x ) is usually achieved by providing more conditional input information in the TTS literature (Wang et al., 2017; Li et al., 2019; Ren et al., 2020) 7 . In this way, more conditional information in input can alleviate the one-to-many mapping issue, and thus the distribution becomes less multimodal and the correlation along time and frequency is reduced given more condition. There are mainly two methods to provide more conditional input information: 1) autoregressive factorization along the time (Wang et al., 2017; Li et al., 2019) or frequency axis; 2) providing more variance information to predict mel-spectrogram in parallel, as used in some non-autoregressive TTS models (Ren et al., 2020; ancucki, 2020). In this section, we first overview these two kinds of methods in detail and conduct the experiment analyses to measure their effectiveness in solving the over-smoothing problem. In this subsection, we overview the two kinds of methods to simplify data distributions, including autoregressive factorization and providing more variance information as input. Autoregressive Factorization The joint probability P ( y | x ) can be factorized according to the chain rule in two ways along time and frequency dimensions respectively:  P ( y | x ) = (cid:81) Tt =1 P ( y t | y <t , x ) , where y <t is the proceeding frames before the t -th frame and T is the total frames. P ( y | x ) = (cid:81) Ff =1 P ( y f | y <f , x ) , where y <f is the proceeding frequency bins before the f -th 7 Ren et al. (2019) use knowledge distillation to simplify the target mel-spectrogram itself, which also simplifies the data distribution but affects data quality as analyzed in Ren et al. (2020). Thus, we do not consider this method in our study. More Variance Information Another way to simplify data distributions P ( y | x ) is to provide more variance information v such as pitch, duration, and energy, to convert P ( y | x ) into P ( y | x, v ) , as used in previous works (Ren et al., 2020; ancucki, 2020). In this way, the distribution becomes less multimodal and the correlation along time and frequency is reduced. Experimental Settings We conduct all experiments 8 on LJSpeech dataset (Ito, 2017). We use ParallelWaveGAN (PWG) (Yamamoto et al., 2020) as the vocoder to convert mel-spectrograms to waveforms. To evaluate the voice quality of the synthesized speech subjectively, we conduct the MOS (Loizou, 2011) tests. To measure the degree of over-smoothness of mel-spectrograms objectively, we calculate the variation of the Laplacian (Pech-Pacheco et al., 2000) (Var L ) on the generated mel-spectrograms and compare with that on the ground-truth mel-spectrograms. We use FastSpeech (Ren et al., 2019) trained with MAE loss as the baseline model as shown in Figure 3. For autoregressive modeling along time, we directly use TransformerTTS (Li et al., 2019). For 8 The corresponding audio samples are available at https: //revisittts.github.io/revisittts/ . autoregressive modeling along frequency, we modify the vanilla mel-spectrogram decoder in baseline model to support autoregressive generation along frequency (which is called FreqAR decoder) and feed y <f to the FreqAR decoder to model P ( y f | y <f ) as shown in Figure 4a. For the method providing more variance information, we use FastSpeech 2 (Ren et al., 2020), which adds pitch, duration and energy information to the variance adaptor of the baseline model. We put more detailed model descriptions in Appendix A and experimental settings in Appendix B. Table 2: Results for different methods for simplifying data distributions for TTS. Results and Analyses We conduct MOS evaluation and compute Var L to compare methods including the baseline model (denoted as MAE ), autoregressive modeling along frequency (denoted as FreqAR ) and along time (denoted as TimeAR ) and FastSpeech 2 . The results are shown in Table 2. We also visualize the mel-spectrograms generated by all modeling methods in Figure 5. From the results, we have some observations: 1) Autoregressive modeling along frequency ( FreqAR ) and time ( TimeAR ) dimensions both outperform MAE in terms of MOS and Var L , which shows that simplifying data distributions using frequency or time dimension factorization can ease the over-smoothing problem. However, the autoregressive models suffer from slow inference. 2) FastSpeech 2 also greatly outperforms the baseline model, further indicating that simplifying data distributions by providing more variance information is another way to alleviate the over-smoothing problem. In conclusion, autoregressive modeling and providing more variance information can both simplify the complex distribution to be less dependent and multimodal and thus alleviate the over-smoothing problem. Besides, methods that provide more variance information such as FastSpeech 2 also enjoy the fast inference speed. Most previous non-autoregressive TTS models (Ren et al., 2019; Wang et al., 2019; Ren et al., 2020) use mean absolute error (MAE) or mean square error (MSE) as training loss. However, they fail to capture dependent and multimodal distributions. MAE loss is derived from the Laplace distribution and MSE from the Gaussian distribution (Chai and Draxler, 2014), which means min-8201 imizing MAE/MSE will maximize the data log-likelihood under a Laplace/Gaussian distribution. Both of these distributions are unimodal and thus encourage the model to predict a single mode in each data point. As a result, the model just learns an average of all modes, which leads to over-smoothed results. Another problem brought by MAE and MSE is that they are independent across time and frequency for mel-spectrogram output, which ignores the correlation across time and frequency axes in mel-spectrogram. In this section, we first introduce several enhanced modeling methods to directly model the dependent and multimodal distribution P ( y | x ) (Sec-tion 4.1), and then conduct experiments to compare and analyze these methods (Section 4.2). We list the enhanced methods, including SSIM loss, Laplacian mixture loss, GAN 9 and Glow-based method and their distribution assumptions in Table 3. We put the details of each method in Appendix A. Structural Similarity Index (SSIM) Structural Similarity Index (SSIM) (Wang et al., 2004) is one of the state-of-the-art perceptual metrics to measure image quality, which can capture structural information and texture. The value of SSIM is between 0 and 1, where 1 indicates perfect perceptual quality relative to the ground truth. The model architecture of SSIM loss follows the baseline model in Figure 3 and we directly replace the MAE loss 9 Although GAN is well-known to suffer from the mode collapse issue, practically it performs very well in modeling the multi-modal distribution through well-tuning (Mao et al., 2017). GAN can avoid the average frame prediction in MAE loss that has a strong unimodal assumption, and can generate high-quality and reasonable results when the data distribution is multimodal. Therefore, we regard GAN as "no distribution assumption" method. Laplacian Mixture (LM) Loss Laplacian mixture loss 10 can model samples independently with multimodal distribution. As shown in Figure 4b, the basic architecture of mel-spectrogram decoder follows baseline model and we modify the output layer of the baseline model to predict the multimodal distribution of each mel-spectrogram bin. Generative Adversarial Network (GAN) We introduce adversarial training to better model the dependent and multimodal distribution. Inspired by Wu and Luan (2020); Binkowski et al. (2020), we adopt multiple random window discriminators. We use the LSGAN (Mao et al., 2017) loss to train the TTS model and multi-window discriminators. Glow Glow (Kingma and Dhariwal, 2018) is a kind of normalizing flow, which maps data into a known and simple prior ( e . g ., spherical multivariate Gaussian distribution). As shown in Figure 4d, our Glow-based decoder models the distribution of mel-spectrograms conditioned on the encoder output hidden states x . 10 We choose Laplace distribution as the mixture distribution since the distribution of the magnitude of spectrogram is Laplacian (Tits et al., 2019; Usman et al., 2018; Gazor and Zhang, 2003). We have also tried other mixture distributions ( e . g ., mixture of logistic and mixture of Gaussian) and have similar findings. The dataset, baseline model and evaluation metrics are the same as Section 3.2. We conduct MOS evaluation and compute Var L to compare different modeling methods including MAE (de-noted as MAE ), Laplacian mixture loss (denoted as LM ), structural similarity index (denoted as SSIM ), generative adversarial network (denoted as GAN ), Glow (denoted as Glow ). The results are shown in Table 4. We also visualize mel-spectrograms generated by all modeling methods in Figure 6. From the results, we can find that: Table 4: Results for different modeling methods for TTS. 1) LM and SSIM outperform MAE in terms of voice quality according to MOS evaluation and the mel-spectrogram visualizations, which shows that even simply replacing the loss function with those without strong unimodal and independent assumptions can significantly alleviate the over-smoothing problem and improve the generated mel-spectrograms. Among simple loss functions, LM performs the best and generates sharper and clearer mel-spectrograms since its Var L is closer to GT , which demonstrates that Laplacian mixture loss can model the multimodal distribution well. 2) GAN and Glow show more superior performances compared with other modeling methods, indicating that modeling mel-spectrogram with both dependent and multimodal distribution can significantly ease the over-smoothing problem and improve the generated speech. From the visualizations, we can see that GAN and Glow generate formants with rich details in the middle/high-frequency region. However, GAN-based and Glow-based methods suffer from training complexity and large model footprint respectively: GAN relies on the discriminator to fit the distribution, which causes unstable training and difficult hyper-parameters tuning; Glow imposes strong architectural constraints and requires a large model footprint (about 2x model parameters) to keep the bijection between the simple independent latent distribution ( e . g ., spherical multivariate Gaussian distribution) and the complex and dependent data distribution. In this section, we first explore the advantages of combining methods from two categories and then perform extensional analyses on multi-speaker dataset. After studying the two categories to combat over-smoothing problems, we have demonstrated that both simplifying data distributions and enhancing modeling methods can alleviate this problem and improve the voice quality of TTS. A natural thought is to combine the methods from two categories, which may integrate the advantages of both aspects to reduce the gap between the complexity of data distribution and the power of the modeling method, resulting in better voice quality and further alleviating the over-smoothing problem. To demonstrate this idea, we choose the FastSpeech 2 as the model from the first category since it achieves better perceptual voice quality than autoregressive modeling models according to Table 2 and importantly, it enjoys the fast and robust inference advantages due to its non-autoregressive nature. Then we combine two categories by applying enhanced modeling methods to FastSpeech 2 and obtain the following systems: 1) FastSpeech 2 + SSIM , which replaces MAE loss with SSIM loss; 2) FastSpeech 2 + LM , which predicts the k-component mixture of Laplace distribution and uses LM loss for training; 3) FastSpeech 2 + GAN , which adds the adversarial loss to FastSpeech 2; and 4) FastSpeech 2 + Glow , which replaces the mel-spectrogram decoder with Glow. We conduct subjective and objective eval-8203 Table 6: Results for of different multi-speaker TTS models on multi-speaker dataset. uations to compare these combined systems with FastSpeech 2 . The results are shown in Table 5. We can see that combining FastSpeech 2 with more powerful modeling methods can further alleviate the over-smoothing problem in terms of Var L and improve the generated speech quality in terms of MOS, which demonstrate our idea that simplifying data distributions and enhancing modeling methods can be used together and it is complementary to each other to further improve TTS performance. Among these methods, GAN performs best in alleviating the over-smoothing problem and Glow achieves the best perceptual voice quality, while they suffer from the cost of increased training or model complexity as we described in Section 4.2; compared with GAN and Glow , LM can generate mel-spectrograms with comparable clearness and quality while enjoying its simplicity. To demonstrate the generalization of our findings and provide more insights, we conduct experiments on a multi-speaker LibriTTS (Zen et al., 2019) dataset. We modify our models to support multiple speakers by adding speaker embeddings to the encoder outputs to indicate the speaker identity. We put more details of our multi-speaker TTS models and the dataset in Appendix C. We compare the following systems: 1) FastSpeech ; 2) FastSpeech + GAN ; 3) FastSpeech + Glow ; 4) FastSpeech 2 ; 5) FastSpeech 2 + GAN ; and 6) FastSpeech 2 + Glow . The results are shown in Table 6. We can see that 1) simplifying data distributions by providing more variance information and enhancing modeling method can alleviate the over-smoothing problem and improve the generated mel-spectrogram, and combining them together can achieve further better audio quality, which are consistent with the findings on single-speaker dataset. 2) FastSpeech + Glow leads to inferior performance compared with the baseline model ( FastSpeech ), because the multi-speaker dataset has more complex target data distributions and Glow requires a large model footprint to capture them as described in Section 4.2. When providing more variance information, FastSpeech 2 + Glow achieves much better performance, which can reduce the difficulty of modeling the target data distribution and thus alleviate the requirements for model capacity. Non-autoregressive Text to Speech Previous TTS systems such as Tacotron (Wang et al., 2017), Tacotron 2 (Shen et al., 2018), Deep Voice 3 (Ping et al., 2018) and TransformerTTS (Li et al., 2019) synthesize speech sequentially, which suffer from slow inference speed. To solve these shortcomings, various non-autoregressive TTS models are proposed to synthesize spectrogram frames in parallel. FastSpeech (Ren et al., 2019) and ParaNet (Peng et al., 2020) are early non-autoregressive TTS model which both adopt a fully parallel model architecture and rely on an autoregressive teacher model to provide the alignment between phonemes and mel-spectrograms. FastSpeech introduces knowledge distillation for mel-spectrograms to simplify data distributions. FastSpeech 2 (Ren et al., 2020) and FastPitch (ancucki, 2020) introduce more variance information as input to further reduce the output uncertainty and ease the one-to-many mapping problem. However, they are trained with MAE loss, which fits independent and unimodal Laplace distribution and results in blurry and over-smoothed mel-spectrograms in inference. SpeedySpeech (Vainer and Duek, 2020) use the combination of MAE and structural similarity index (SSIM) losses to avoid blurry mel-spectrograms. Glow-TTS (Kim et al., 2020) and Flow-TTS (Miao et al., 2020) both use flow-based decoder to apply some invertible transforms between mel-spectrograms and noise data sampled from simple distribution. Sheng and Pavlovskiy (2019) employ a cascaded Tacotron 2 and GAN pipeline to reduce the over-smoothness of synthesized speech. Multi-SpectroGAN (Lee et al., 2020) introduces generative adversarial network (GAN) and a multi-scale discriminator and is trained with only the adversarial feedback by conditioning hidden states with variance information ( e . g ., duration, 8204 pitch and energy) to a discriminator. GAN and Flow-based methods can model dependent and multimodal distribution well, while they suffer from training or model complexity. In this work, we conduct systematic studies on several modeling methods in both NAR-TTS and AR-TTS from a novel perspective. Handling Dependent and Multimodal Distributions Dependent and multimodal distributions increase the uncertainty for model training and lead to blurry results, which is observed in many generation tasks (Gu et al., 2018; Isola et al., 2017; Mathieu et al., 2016). There are some common ways to handle dependent and multimodal distributions: 1) using loss functions or modeling methods that can well fit the distributions; and 2) introducing some input variables or transforming the target data to simplify data distributions. In neural machine translation, Gu et al. (2018) tackle this problem by introducing knowledge distillation to simplify target data distributions and using fertilities extracted by an external aligner to directly model the nondeterminism in the translation process. In image translation task, Isola et al. (2017) compare the generated images by their proposed adversarial generative method with those generated by using MAE and MSE losses and conclude that adversarial loss can help avoid blurry results. In video prediction task, Mathieu et al. (2016) propose a multi-scale architecture, an adversarial training method, and an image gradient difference loss function to deal with the inherently blurry predictions obtained from the MSE loss function. However, there is no systematic analysis on multimodal and dependent distributions in TTS task as far as we know. In this paper, we conduct comprehensive analyses and studies on handling dependent and multimodal distributions in TTS from a novel perspective. In this paper, we revisited the over-smoothing problem in TTS with a novel perspective: the degree of over-smoothness is determined by the gap between the complexity of data distribution and the capability of the modeling method. Under this perspective, we classified existing methods combating over-smoothness into two categories: simplifying data distributions and enhancing modeling methods, and conducted comprehensive analyses and studies on these methods. For simplifying data distributions, we found that both AR factorization and providing more variance information as input ( e . g ., FastSpeech 2) can alleviate the over-smoothing problem, and FastSpeech 2 enjoys the advantage of fast generation over AR factorization. For enhancing modeling methods, we found that Laplacian mixture loss can improve the generation quality and enjoy its simplicity, while GAN and Glow can further achieve better quality at the cost of increased training or model complexity. Based on the above findings, we further combined these two categories of methods and found that the over-smoothing problem is further alleviated, and the generated speech quality is further improved, which shows that these two categories are complementary to each other. When performing our analyses on the multi-speaker dataset, we drew similar conclusions and found that providing more variance information can reduce the difficulty of modeling the target data distribution and alleviate the requirements for model capacity. We hope that our studies can inspire the community and industry to develop more powerful TTS models. Besides, since we are the first to discuss the over-smoothing problem systematically in the speech domain. We hope our analysis methodology as well as the findings can be extended to other tasks and inspire other domains. This work was supported in part by Zhejiang Natural Science Foundation under Grant LR19F020006, National Natural Science Foundation of China under Grant No.61836002, No.62072397.
abstain abstain abstain objective abstain objective method result abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method result abstain abstain abstain objective method abstain abstain abstain method method objective objective abstain objective objective objective result objective result abstain abstain abstain abstain method method abstain abstain method other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain result other result abstain result result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain method abstain method abstain method method method method abstain method method method abstain method method abstain abstain abstain method other abstain method abstain method method method abstain method method abstain method method abstain abstain abstain other other other other other other other other other other other objective other other other other other abstain objective objective method result result result result objective objective objective other