summaries: This submission belongs to the field of text-to-speech synthesis. In particular it looks at a novel way of formulating a normalising flow using 2D rather than conventional 1D representation. Such reformulation enables to provide interpretations to several existing approaches as well as formulate a new one with quite interesting properties. This submission would benefit from a discussion of limitations of your approach.  This paper re-organized the high dimensional 1-D raw waveform as 2-D matrix. This method simulated the autoregressive flow. Log-likelihood could be calculated in parallel. Autoregressive flow was only run on row dimension. The number of required parameters was desirable to synthesize high-fidelity speech with the speed faster than real time. Although this method could not achieve top one in ranking in every measurements, the resulting performance was still obtained with the best average results. 
abstractText: In this work, we present WaveFlow, a small-footprint generative flow for raw audio, which is trained with maximum likelihood without density distillation and auxiliary losses as used in Parallel WaveNet. It provides a unified view of flowbased models for raw audio, including autoregressive flow (e.g., WaveNet) and bipartite flow (e.g., WaveGlow) as special cases. We systematically study these likelihood-based generative models for raw waveforms in terms of test likelihood and speech fidelity. We demonstrate that WaveFlow can synthesize high-fidelity speech and obtain comparable likelihood as WaveNet, while only requiring a few sequential steps to generate very long waveforms. In particular, our small-footprint WaveFlow has 5.91M parameters and can generate 22.05kHz high-fidelity speech 42.6× faster than real-time on a GPU without engineered inference kernels. 1
references.authors: S.O. Arık M. Chrzanowski A. Coates G. Diamos A. Gibiansky Y. Kang X. Li J. Miller J. Raiman S. Sengupta M. Shoeybi
references.title: Deep Voice: Real-time neural text-to-speech
references.venue: In ICML,
references.authors: S.O. Arık G. Diamos A. Gibiansky J. Miller K. Peng W. Ping J. Raiman Y. Zhou
references.title: Deep Voice 2: Multi-speaker neural text-to-speech
references.venue: In NIPS,
references.authors: R. v. d. Berg L. Hasenclever J.M. Tomczak M. Welling
references.title: Sylvester normalizing flows for variational inference
references.venue: arXiv preprint arXiv:1803.05649,
references.authors: S. Dieleman A. van den Oord K. Simonyan
references.title: The challenge of realistic music generation: modelling raw audio at scale
references.venue: NeurIPS,
references.authors: L. Dinh D. Krueger Y. Bengio
references.title: NICE: Non-linear independent components estimation
references.venue: arXiv preprint arXiv:1410.8516,
references.authors: L. Dinh J. Sohl-Dickstein S. Bengio
references.title: Density estimation using Real NVP
references.venue: In ICLR,
references.authors: W. Grathwohl R.T. Chen J. Betterncourt I. Sutskever D. Duvenaud
references.title: Ffjord: Free-form continuous dynamics for scalable reversible generative models
references.venue: arXiv preprint arXiv:1810.01367,
references.authors: E. Hoogeboom R. v. d. Berg M. Welling
references.title: Emerging convolutions for generative normalizing flows
references.venue: arXiv preprint arXiv:1901.11137,
references.authors: C.-W. Huang D. Krueger A. Lacoste A. Courville
references.title: Neural autoregressive flows
references.venue: arXiv preprint arXiv:1804.00779,
references.authors: K. Ito
references.title: The LJ speech dataset. 2017
references.authors: N. Kalchbrenner E. Elsen K. Simonyan S. Noury N. Casagrande E. Lockhart F. Stimberg A. v. d. Oord S. Dieleman K. Kavukcuoglu
references.title: Efficient neural audio synthesis
references.authors: S. Kim S.-g. Lee J. Song S. Yoon
references.title: FloWaveNet: A generative flow for raw audio
references.venue: In ICML,
references.authors: D.P. Kingma J. Ba
references.title: Adam: A method for stochastic optimization
references.venue: In ICLR,
references.authors: D.P. Kingma P. Dhariwal
references.title: Glow: Generative flow with invertible 1x1 convolutions
references.venue: In Advances in Neural Information Processing Systems,
references.authors: D.P. Kingma T. Salimans R. Jozefowicz X. Chen I. Sutskever M. Welling
references.title: Improving variational inference with inverse autoregressive flow
references.authors: N. Li S. Liu Y. Liu S. Zhao M. Liu M. Zhou
references.title: Neural speech synthesis with transformer network
references.authors: S. Mehri K. Kumar I. Gulrajani R. Kumar S. Jain J. Sotelo A. Courville Y. Bengio
references.title: SampleRNN: An unconditional end-to-end neural audio generation model
references.venue: In ICLR,
references.authors: T.L. Paine P. Khorrami S. Chang Y. Zhang P. Ramachandran M.A. Hasegawa-Johnson T.S. Huang
references.title: Fast wavenet generation algorithm
references.venue: arXiv preprint arXiv:1611.09482,
references.authors: G. Papamakarios T. Pavlakou I. Murray
references.title: Masked autoregressive flow for density estimation
references.venue: In Advances in Neural Information Processing Systems,
references.authors: K. Peng W. Ping Z. Song K. Zhao
references.title: Parallel neural text-to-speech
references.venue: arXiv preprint arXiv:1905.08459,
references.authors: B. Pharris
references.title: NV-WaveNet: Better speech synthesis using gpu-enabled WaveNet inference
references.venue: In NVIDIA Developer Blog,
references.authors: W. Ping K. Peng A. Gibiansky S.O. Arik A. Kannan S. Narang J. Raiman J. Miller
references.title: Deep Voice 3: Scaling text-to-speech with convolutional sequence learning
references.venue: In ICLR,
references.authors: W. Ping K. Peng J. Chen
references.title: ClariNet: Parallel wave generation in end-to-end text-to-speech
references.venue: In ICLR,
references.authors: R. Prenger R. Valle B. Catanzaro
references.title: WaveGlow: A flow-based generative network for speech synthesis
references.venue: In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),
references.authors: Y. Ren Y. Ruan X. Tan T. Qin S. Zhao Z. Zhao T.-Y. Liu
references.title: Fastspeech: Fast, robust and controllable text to speech
references.authors: D.J. Rezende S. Mohamed
references.title: Variational inference with normalizing flows
references.venue: In ICML,
references.authors: F. Ribeiro D. Florêncio C. Zhang M. Seltzer
references.title: CrowdMOS: An approach for crowdsourcing mean opinion score studies
references.venue: In ICASSP,
references.authors: T. Salimans D.P. Kingma
references.title: Weight normalization: A simple reparameterization to accelerate training of deep neural networks
references.venue: In Advances in Neural Information Processing Systems,
references.authors: J. Serrà S. Pascual C. Segura
references.title: Blow: a single-scale hyperconditioned flow for non-parallel raw-audio voice conversion
references.venue: arXiv preprint arXiv:1906.00794,
references.authors: J. Shen R. Pang R.J. Weiss M. Schuster N. Jaitly Z. Yang Z. Chen Y. Zhang Y. Wang R. SkerryRyan
references.title: Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions
references.authors: J. Sotelo S. Mehri K. Kumar J.F. Santos K. Kastner A. Courville Y. Bengio
references.title: Char2wav: End-to-end speech synthesis
references.venue: ICLR workshop,
references.authors: Y. Taigman L. Wolf A. Polyak E. Nachmani
references.title: VoiceLoop: Voice fitting and synthesis via a phonological loop
references.venue: In ICLR,
references.authors: A. van den Oord S. Dieleman H. Zen K. Simonyan O. Vinyals A. Graves N. Kalchbrenner A. Senior K. Kavukcuoglu
references.title: WaveNet: A generative model for raw audio
references.venue: arXiv preprint arXiv:1609.03499,
references.authors: A. van den Oord Y. Li I. Babuschkin K. Simonyan O. Vinyals K. Kavukcuoglu G. v. d. Driessche E. Lockhart L.C. Cobo F. Stimberg
references.title: Parallel WaveNet: Fast high-fidelity speech synthesis
references.authors: Y. Wang R. Skerry-Ryan D. Stanton Y. Wu R.J. Weiss N. Jaitly Z. Yang Y. Xiao Z. Chen S. Bengio Q. Le Y. Agiomyrgiannakis R. Clark R.A. Saurous
references.title: Tacotron: Towards end-to-end speech synthesis
references.authors: R. Yamamoto E. Song J.-M. Kim
references.title: Probability density distillation with generative adversarial networks for high-quality parallel waveform generation
references.authors: F. Yu V. Koltun
references.title: Multi-scale context aggregation by dilated convolutions
references.venue: arXiv preprint arXiv:1511.07122,
sections.heading: 1 INTRODUCTION
sections.text: Deep generative models have obtained noticeable successes for modeling raw audio in high-fidelity speech synthesis and music generation (e.g., van den Oord et al., 2016; Dieleman et al., 2018). Autoregressive models are among the best performing generative models for raw audio waveforms, providing the highest likelihood scores and generating high quality samples (e.g., van den Oord et al., 2016; Kalchbrenner et al., 2018). One of the most successful examples is WaveNet (van den Oord et al., 2016), an autoregressive model for waveform synthesis. It operates at the high temporal resolution of raw audio (e.g., 24kHz) and sequentially generates waveform samples at inference. As a result, WaveNet is prohibitively slow for speech synthesis and one has to develop highly engineered kernels for real-time inference (Arık et al., 2017a; Pharris, 2018). 2
  Flow-based models (Dinh et al., 2014; Rezende and Mohamed, 2015) are a family of generative models, in which a simple initial density is transformed into a complex one by applying a series of invertible transformations. One group of models are based on autoregressive transformation, including autoregressive flow (AF) and inverse autoregressive flow (IAF) as the “dual” of each other (Kingma et al., 2016; Papamakarios et al., 2017; Huang et al., 2018). AF is analogous to autoregressive models, which performs parallel density evaluation and sequential synthesis. In contrast, IAF performs parallel synthesis but sequential density evaluation, making likelihood-based training very slow. Parallel WaveNet (van den Oord et al., 2018) distills an IAF from a pretrained autoregressive WaveNet, which gets the best of both worlds. However, it requires the density distillation with Monte Carlo approximation and a set of auxiliary losses for good performance, which complicates the training pipeline and increases the cost of development. Instead, ClariNet (Ping et al., 2019) simplifies the density distillation by computing a regularized KL divergence in closed-form.
  Another group of flow-based models are based on bipartite transformation (Dinh et al., 2017; Kingma and Dhariwal, 2018), which provide parallel density evaluation and parallel synthesis. Most recently, WaveGlow (Prenger et al., 2019) and FloWaveNet (Kim et al., 2019) successfully applies Glow (Kingma and Dhariwal, 2018) and RealNVP (Dinh et al., 2017) for waveform synthesis, respectively. However, the bipartite transformations are less expressive than the autoregressive transformations (see Section 2.3 for detailed discussion). In general, these bipartite flows require
  1Audio samples are located at: https://waveflow-demo.github.io/. 2Real-time inference is a requirement for most production text-to-speech systems. For example, if the system
  can synthesize 1 second of speech in 0.5 seconds, it is 2× faster than real-time.
  deeper layers, larger hidden size, and huge number of parameters to reach comparable capacities as autoregressive models. For example, WaveGlow and FloWaveNet have 87.88M and 182.64M parameters with 96 layers and 256 residual channels, respectively. In contrast, a 30-layer WaveNet has only 4.57M parameters with 128 residual channels.
  In this work, we present WaveFlow, a compact flow-based model for raw audio. Specifically, we make the following contributions:
  1. WaveFlow is trained with maximum likelihood without density distillation and auxiliary losses used in Parallel WaveNet (van den Oord et al., 2018) and ClariNet (Ping et al., 2019), which simplifies the training pipeline and reduces the cost of development.
  2. WaveFlow squeezes the 1-D raw waveforms into a 2-D matrix and produces the whole audio within a fixed sequential steps. It also provides a unified view of flow-based models for raw audio and allows us to explicitly trade inference efficiency for model capacity. We implement WaveFlow with a dilated 2-D convolutional architecture (Yu and Koltun, 2015), and it includes both Gaussian WaveNet (Ping et al., 2019) and WaveGlow (Prenger et al., 2019) as special cases.
  3. We systematically study the likelihood-based generative models for raw audios in terms of test likelihood and speech quality. We demonstrate that WaveFlow can obtain comparable likelihood and synthesize high-fidelity speech as WaveNet (van den Oord et al., 2016), while only requiring a few sequential steps to generate very long waveforms.
  4. Our small-footprint WaveFlow has only 5.91M parameters and synthesizes 22.05 kHz highfidelity speech (MOS: 4.32) more than 40× faster than real-time on a Nvidia V100 GPU. In contrast, WaveGlow (Prenger et al., 2019) requires 87.8M parameters for generating high-fidelity speech. The small memory footprint is preferred in production TTS systems, especially for on-device deployment.
  We organize the rest of the paper as follows. Section 2 reviews the flow-based models with autoregressive and bipartite transformations. We present WaveFlow in Section 3 and discuss related work in Section 4. We report experimental results in Section 5 and conclude the paper in Section 6.
sections.heading: 2 FLOW-BASED GENERATIVE MODELS
sections.text: Flow-based models (Dinh et al., 2014; 2017; Rezende and Mohamed, 2015) transform a simple density of latent variables p(z) (e.g., isotropic Gaussian) into a complex data distribution p(x) by applying a bijection x = f(z), where x and z are both n-dimensional. The probability density of x can be obtained through the change of variables formula:
  p(x) = p(z) ∣∣∣∣det(∂f−1(x)∂x )∣∣∣∣ , (1)
  where z = f−1(x) is the inverse transformation, and det (∂f−1(x)
  ∂x
  ) is the determinant of its Jacobian.
  In general, it takes O(n3) to compute the determinant, which is not scalable to high-dimensional data. There are two notable groups of flow-based models with triangular Jacobians and tractable determinants. They are based on autoregressive and bipartite transformations, respectively.
sections.heading: 2.1 AUTOREGRESSIVE TRANSFORMATION
sections.text: The autoregressive flow (AF) and inverse autoregressive flow (IAF) (Kingma et al., 2016; Papamakarios et al., 2017) use autoregressive transformations. Specifically, AF defines the inverse transformation z = f−1(x;ϑ) as:
  zt = xt · σt(x<t;ϑ) + µt(x<t;ϑ), (2)
  where the shifting variables µt(x<t;ϑ) and scaling variables σt(x<t;ϑ) are modeled by an autoregressive architecture parameterized by ϑ (e.g., WaveNet). Note that, the t-th variable zt only depends on x≤t, thus the Jacobian is a triangular matrix as illustrated in Figure 1(a) and its determinant
  is the product of the diagonal entries: det ( ∂f−1(x) ∂x ) = ∏ t σt(x<t;ϑ). The density p(x) can be
  easily evaluated by change of variables formula, because z = f−1(x) can be computed in parallel from Eq. (2) (i.e., the required O(n) operations can be done in O(1) time on modern GPU hardware). However, AF has to do sequential synthesis, because the forward transformation x = f(z) is autoregressive: xt =
  zt−µt(x<t;ϑ) σt(x<t;ϑ)
  . In contrast, IAF uses an autoregressive transformation for z = f−1(x):
  zt = xt − µt(z<t;ϑ) σt(z<t;ϑ) , (3)
  making density evaluation impractically slow for training, but it can do parallel synthesis by xt = zt · σt(z<t;ϑ) + µt(z<t;ϑ). Parallel WaveNet (van den Oord et al., 2018) and ClariNet (Ping et al., 2019) are based on IAF, which lacks efficient density evaluation and relies on distillation from a pretrained autoregressive WaveNet.
sections.heading: 2.2 BIPARTITE TRANSFORMATION
sections.text: RealNVP (Dinh et al., 2017) and Glow (Kingma and Dhariwal, 2018) use bipartite transformation by partitioning the data x into two groups xa and xb, where the indices sets a ∪ b = {1, · · · , n} and a ∩ b = φ. Then, the inverse transformation z = f−1(x,θ) is defined as:
  za = xa, zb = xb · σb(xa;θ) + µb(xa;θ). (4)
  where the shifting variables µb(xa;θ) and scaling variables σb(xa;θ) are modeled by a feed-forward neural network. The Jacobian ∂f
  −1(x) ∂x is a special triangular matrix as illustrated in Figure 1 (b). By
  definition, the forward transformation x = f(z,θ) is,
  xa = za, xb = zb − µb(xa;θ) σb(xa;θ) , (5)
  and can also be done in parallel. As a result, the bipartite transformation provides both parallel density evaluation and parallel synthesis. In previous work, WaveGlow (Prenger et al., 2019) and FloWaveNet (Kim et al., 2019) both squeeze the adjacent audio samples on the channel dimension, and apply the bipartite transformation on the partitioned channel dimension.
sections.heading: 2.3 CONNECTIONS
sections.text: It is worthwhile to mention that the autoregressive transformation is more expressive than bipartite transformation in general. As illustrated in Figure 1(a) and (b), the autoregressive transformation introduces n×(n−1)2 complex non-linear dependencies (dark-blue cells) and n linear dependencies between data x and latents z. In contrast, bipartite transformation introduces only n 2
  4 non-linear
  dependencies and n2 linear dependencies. Indeed, one can reduce an autoregressive transformation z = f−1(x;ϑ) to a bipartite transformation z = f−1(x;θ) by: (i) picking an autoregressive order o such that all of the indices in set a rank early than the indices in b, and (ii) setting the shifting and scaling variables as,
  µt(x<t;ϑ) = { 0 for t ∈ a µt(xa;θ) for t ∈ b , σt(x<t;ϑ) = { 1 for t ∈ a σt(xa;θ) for t ∈ b .
  Given the less expressive building block, the bipartite transformation-based flows generally require many more layers and larger hidden size to match the capacity of a compact autoregressive models (e.g., as measured by test likelihood) (Kingma and Dhariwal, 2018; Prenger et al., 2019).
sections.heading: 3 WAVEFLOW
sections.text: In this section, we present WaveFlow and its implementation with dilated 2-D convolutions.
sections.heading: 3.1 DEFINITION
sections.text: We denote the high dimensional 1-D waveform as x = {x1, · · · , xn}. We first squeeze x into a h-row 2-D matrix X ∈ Rh×w by column-major order, where w = nh and adjacent samples are in the same column. We assume Z ∈ Rh×w are sampled from an isotropic Gaussian, and define the inverse transformation Z = f−1(X; Θ) as,
  Zi,j = σi,j(X<i,•; Θ) ·Xi,j + µi,j(X<i,•; Θ), (6)
  where X<i,• represents all elements above i-th row (see Figure 2 for an illustration). Note that, i) the receptive fields over the squeezed inputs X for computing Zi,j in WaveFlow is strictly larger than that of WaveGlow when h > 2. ii) WaveNet is equivalent to an autoregressive flow with column-major order on the squeezed inputs X . iii) Both WaveFlow and WaveGlow look at future waveform samples in original x for computing Zi,j , whereas WaveNet can not. iv) The autoregressive flow with row-major order has larger receptive fields than WaveFlow and WaveGlow.
  The shifting variables µi,j(X<i,•; Θ) and scaling variables σi,j(X<i,•; Θ) in Eq. (6) are modeled by a 2-D convolutional neural network detailed in Section 3.2. By definition, the variable Zi,j only depends on the current Xi,j and previous X<i,• in raw-major order, thus the Jacobian is a triangular matrix and its determinant is:
  det
  ( ∂f−1(X)
  ∂X
  ) = h∏ i=1 w∏ j=1 σi,j(X<i,•; Θ). (7)
  As a result, the log-likelihood can be calculated in parallel by change of variable formula in Eq. (1),
  log p(X) = − h∑ i=1 w∑ j=1 ( Z2i,j + 1 2 log(2π) ) + h∑ i=1 w∑ j=1 log σi,j(X<i,•; Θ), (8)
  and one can do maximum likelihood training efficiently. At synthesis, one may first sample Z from the isotropic Gaussian and apply the forward transformation X = f(Z; Θ):
  Xi,j = Zi,j − µi,j(X<i,•; Θ)
  σi,j(X<i,•; Θ) , (9)
  which is only autoregressive on height dimension. Thus, it requires h sequential steps to generate the whole waveform X . In practice, a small h (e.g., 8 or 16) works well, thus we can generate very long waveforms within a few sequential steps.
sections.heading: 3.2 IMPLEMENTATION WITH DILATED 2-D CONVOLUTIONS
sections.text: In this work, we implement WaveFlow with a dilated 2-D convolutional architecture. Specifically, we use a stack of 2-D convolution layers (e.g., 8 layers in all experiments) to model the shifting variables µi,j(X<i,•; Θ) and scaling variables σi,j(X<i,•; Θ) in Eq. (6). We use the similar architecture as
  WaveNet (van den Oord et al., 2016) by replacing the dilated 1-D convolution to 2-D convolution (Yu and Koltun, 2015), while still keeping the gated-tanh nonlinearities, residual connections and skip connections.
  We set the filter sizes as 3 for both height and width dimensions. We use non-causal convolutions on width dimension and set the dilation cycle as [1, 2, 4, · · · , 27]. The convolutions on height dimension are causal with an autoregressive constraint, and their dilation cycle needs to be designed carefully. In practice, we find the following rules of thumb are important to obtain good results:
  • As motivated by the dilation cycle of WaveNet (van den Oord et al., 2016), the dilations of 8 layers should be set as d = [1, 2, · · · , 2s, 1, 2, · · · , 2s, · · · ], where s ≤ 7. 3
  • The receptive field r over the height dimension should be larger than the squeezed height h. Otherwise, it explicitly introduces unnecessary conditional independence and leads to lower likelihood (see Table 1 for an example). Note that, the receptive field of a stack of dilated convolutional layers is: r = (k−1)× ∑ i di+1, where k is the filter size and di is the dilation
  at i-th layer. Thus, the sum of dilations should satisfy: ∑ i di ≥ h−1 k−1 . However, when h is
  larger than or equal to 28 = 512, we simply set the dilation cycle as [1, 2, 4, · · · , 27]. • When the receptive field r has already been larger than h, we find that convolutions with
  smaller dilation and fewer holes provide larger likelihood.
  We summarize the heights and preferred dilations in our experiments in Table 2. Note that, WaveFlow becomes fully autoregressive when we squeeze x by its length (i.e. h = n) and set its filter size as 1 over the width dimension, which is equivalent to a Gaussian WaveNet learned by MLE (Ping et al., 2019). If we squeeze x by h = 2 and set the filter size as 1 on the height dimension, WaveFlow becomes a bipartite flow and is equivalent to WaveGlow with squeezed channels 2.
sections.heading: 3.3 CONDITIONAL GENERATION
sections.text: In neural speech synthesis, a neural vocoder (e.g., WaveNet) synthesizes the time-domain waveforms. It can be conditioned on linguistic features (van den Oord et al., 2016; Arık et al., 2017a), the mel-spectrograms from a text-to-spectrogram model (Ping et al., 2018; Shen et al., 2018), or the
  3We did try different setups, but they all lead to worse likelihood scores.
  learned hidden representation within a text-to-wave architecture (Ping et al., 2019). In this work, we test WaveFlow by conditioning it on ground truth mel-spectrograms as in previous work (Prenger et al., 2019; Kim et al., 2019). The mel-spectrogram is upsampled to the same resolution as waveform samples by transposed 2-D convolutions (Ping et al., 2019). To aligned with the squeezed waveform, they are squeezed to the shape c × h × w, where c is the feature dimension (e.g, bands of the spectrogram). After a 1× 1 convolution mapping the features to residual channels, they are added as the bias term at each layer (van den Oord et al., 2016).
sections.heading: 3.4 STACKING MULTIPLE FLOWS WITH PERMUTATIONS OVER HEIGHT DIMENSION
sections.text: Flow-based models require a series of transformations until the distribution p(X) reaches a desired level of complexity (e.g., Rezende and Mohamed, 2015). We let X = Z(n) and repeatedly apply the transformation Z(i−1) = f−1(Z(i); Θ(i)) defined in Eq. (6) from Z(n) → . . . Z(i) → . . . Z(0). We assume Z(0) is from the isotropic Gaussian distribution. The likelihood p(X) can be evaluated by iteratively applying the chain rule:
  p(X) = p(Z(0)) n∏ i=1 ∣∣∣∣det(∂f−1(Z(i); Θ(i))∂Z(i) )∣∣∣∣ .
  We find that permuting each Z(i) over the height dimension after each transformation can significantly improve the likelihood scores. In particular, we test two permutation strategies for WaveFlow models stacked with 8 flows (i.e., X = Z(8)) in Table 3: (i) we reverse each Z(i) over the height dimension after each transformation, and (ii) we reverse Z(7), Z(6), Z(5), Z(4) over the height dimension as before, but split Z(3), Z(2), Z(1), Z(0) in the middle of the height dimension then reverse each part respectively. 4 Note that, one also needs to permute the conditioner on the height dimension accordingly, which is aligned with Z(i). From Table 3, both (i) and (ii) significantly outperform the model without permutations mainly because of bidirectional modeling. Strategy (ii) outperforms (i) because of its diverse autoregressive orders.
sections.heading: 4 RELATED WORK
sections.text: Deep neural networks for speech synthesis (a.k.a. text-to-speech) have received a lot of attention. Over the past few years, several neural text-to-speech (TTS) systems have been introduced, including WaveNet (van den Oord et al., 2016), Deep Voice (Arık et al., 2017a), Deep Voice 2 (Arık et al., 2017b), Deep Voice 3 (Ping et al., 2018), Tacotron (Wang et al., 2017), Tacotron 2 (Shen et al., 2018), Char2Wav (Sotelo et al., 2017), VoiceLoop (Taigman et al., 2018), WaveRNN (Kalchbrenner et al., 2018), ClariNet (Ping et al., 2019), Transformer TTS (Li et al., 2019), ParaNet (Peng et al., 2019) and FastSpeech (Ren et al., 2019).
  Neural vocoders, such as WaveNet, play the most important role in recent advances of speech synthesis. In previous work, the state-of-the-art neural vocoders are autoregressive models (van den Oord et al., 2016; Mehri et al., 2017; Kalchbrenner et al., 2018). Several engineering endeavors have been advocated for speeding up their sequential generation process (Arık et al., 2017a; Kalchbrenner et al., 2018). In particular, Subscale WaveRNN (Kalchbrenner et al., 2018) folds a long waveform sequence x1:n into a batch of shorter sequences and can produces up to 16 samples per step, thus it requires at least n16 steps to generate the whole audio. Note that, this is different from the proposed WaveFlow, which can generate x1:n within a fixed number of steps (e.g., 16). Most recently, flowbased models have been successfully applied for parallel waveform synthesis with comparable fidelity
  4After split & reverse operations, the height dimension [0, · · · , h 2 − 1, h 2 , · · · , h− 1] becomes [h 2 − 1, · · · ,
  0, h− 1, · · · , h 2 ].
  as autoregressive models (van den Oord et al., 2018; Ping et al., 2019; Prenger et al., 2019; Kim et al., 2019; Yamamoto et al., 2019; Serrà et al., 2019). Among these models, WaveGlow (Prenger et al., 2019) and FloWaveNet (Kim et al., 2019) have a simple training pipeline as they solely use the maximum likelihood objective. However, both of them are less expressive than autoregressive models as indicated by their lower likelihood scores.
  Flow-based models can either represent the approximate posteriors for variational inference (Rezende and Mohamed, 2015; Kingma et al., 2016; Berg et al., 2018), or can be trained directly on data using the change of variables formula (Dinh et al., 2014; 2017; Kingma and Dhariwal, 2018; Grathwohl et al., 2018). In previous work, Glow (Kingma and Dhariwal, 2018) extends RealNVP (Dinh et al., 2017) with invertible 1× 1 convolution, and can generate high quality images. Later on, Hoogeboom et al. (2019) generalizes the 1× 1 convolution to invertible d× d convolutions which operate both channel and spatial axes.
sections.heading: 5 EXPERIMENT
sections.text: In this section, we compare likelihood-based generative models for raw audio in term of test likelihood, speech quality and synthesis speed.
  Data: We use the LJ speech dataset (Ito, 2017) containing about 24 hours of audio with a sampling rate of 22.05kHz recorded on a MacBook Pro in a home enviroment. It consists of 13, 100 audio clips of a single female speaker reading passages from 7 non-fiction books.
  Models: We evaluate several likelihood-based generative models, including Gaussian WaveNet, WaveGlow, WaveFlow and autoregressive flow (AF). As in Section 3.2, we implement autoregressive flow from WaveFlow by squeezing the waveforms by its length and setting the filter size as 1 for width dimension. Both WaveNet and AF have 30 layers with dilation cycle [1, 2, · · · , 512] and filter size 3. For WaveGlow and WaveFlow, we investigate different setups, including the number of flows, size of residual channels, and squeezed height h.
  Conditioner: We use the 80-band mel-spectrogram of the original audio as the conditioner for WaveNet, WaveGlow, and WaveFlow. We use FFT size 1024, hop size 256, and window size 1024. For WaveNet and WaveFlow, we upsample the mel conditioner 256 times by applying two layers of transposed 2-D convolution (in time and frequency) interleaved with leaky ReLU (α = 0.4). The upsampling strides in time are 16 and the 2-D convolution filter sizes are [32, 3] for both layers. For WaveGlow, we directly use the open source implementation. 5
  Training: We train all models on 8 Nvidia 1080Ti GPUs using randomly chosen short clips of 16, 000 samples from each utterance. For WaveFlow and WaveNet, we use the Adam optimizer (Kingma and Ba, 2015) with a batch size of 8 and a constant learning rate of 2× 10−4. For WaveGlow, we use the Adam optimizer with a batch size of 16 and a learning rate of 1× 10−4. We applied weight normalization (Salimans and Kingma, 2016) whenever possible.
sections.heading: 5.1 LIKELIHOOD
sections.text: The test log-likelihoods (LLs) of all models are evaluate at 1M training steps. Note that, i) all of the LLs decrease slowly after 1M steps and ii) it took one month to train the largest WaveGlow (residual channels = 512) for 1M steps. Thus, we chose 1M as the cut-off to compare these models. We summarize the results in Table 4 with models from row (a) to (t). We draw the following observations:
  • Stacking a large number of flows improves LLs for WaveFlow, autoregressive flow, and WaveGlow. For example, (m) WaveFlow with 8 flows provide larger LL than (l) WaveFlow with 6 flows. The (b) autoregressive flow obtains the highest likelihood and even outperforms (a) WaveNet with the same amount of parameters. Indeed, AF provides bidirectional modeling by stacking 3 flows interleaved with reverse operations.
  • WaveFlow has much larger likelihood than WaveGlow with comparable number of parameters. In particular, a small-footprint (k) WaveFlow has only 5.91M parameters but can provide comparable likelihood (5.023 vs. 5.026) as the largest (g) WaveGlow with 268.29M parameters.
  5https://github.com/NVIDIA/waveglow
sections.heading: 5.2 SPEECH FIDELITY AND SYNTHESIS SPEED
sections.text: We train WaveNet for 1M steps. We train WaveGlow and WaveFlow for 2M steps with small residual channels (64, 96 and 128). We train larger models (res. channels 256 and 512) for 1M steps due to the practical time constraint. At synthesis, we sampled Z from an isotropic Gaussian with standard deviation 1.0 and 0.6 (default) for WaveFlow and WaveGlow, respectively. For WaveFlow and WaveGlow, we run synthesis under NVIDIA Apex with 16-bit floating point (FP16) arithmetic, which does not introduce any degradation of audio fidelity and brings about 2× speedup. We use the crowdMOS tookit (Ribeiro et al., 2011) for naturalness evaluation, where test utterances from these models were presented to workers on Mechanical Turk. We also test the synthesis speed on a Nvidia V100 GPU without using any customized inference kernels. We only implement convolution queues (Paine et al., 2016) in Python to cache the intermediate hidden states within WaveFlow for autoregressive inference over the height dimension, which brings about 4× speedup. We use the permutation strategy (ii) described in Section 3.4 for WaveFlow.
  We report the 5-scale Mean Opinion Score (MOS), synthesis speed and model footprint in Table 5. We draw the following observations:
  • The small WaveFlow (res. channels 64) has 5.91M parameters and can synthesize 22.05 kHz high-fidelity speech (MOS: 4.32) 42.60× faster than real-time. In contrast, the speech quality of small WaveGlow (res. channels 64) is significantly worse (MOS: 2.17). Indeed, WaveGlow (res. channels 256) requires 87.88M parameters for generating high-fidelity speech.
  • The large WaveFlow (res. channels 256) outperforms the same size WaveGlow in terms of speech fidelity (MOS: 4.43 vs. 4.34). It also matches the state-of-the-art WaveNet while gener-
sections.heading: 6 CONCLUSION
sections.text: We propose WaveFlow, a compact flow-based model for raw audio, which can be directly trained with maximum likelihood estimation. It provides a unified view of flow-based models for time-domain waveforms, and includes WaveNet and WaveGlow as special cases. WaveFlow requires a small number of sequential steps to generate high-fidelity speech and obtains likelihood comparable to WaveNet. In the end, our small-footprint WaveFlow can generate 22.05kHz high-fidelity speech more than 40× faster than real-time on a GPU without engineered inference kernels.
title: WAVEFLOW: A COMPACT FLOW-BASED MODEL