HIERARCHICAL GENERATIVE MODELING FOR CONTROLLABLE SPEECH SYNTHESIS
This paper proposes a neural sequence-to-sequence text-to-speech (TTS) model which can control latent attributes in the generated speech that are rarely annotated in the training data, such as speaking style, accent, background noise, and recording conditions. The model is formulated as a conditional generative model based on the variational autoencoder (VAE) framework, with two levels of hierarchical latent variables. The first level is a categorical variable, which represents attribute groups (e.g. clean/noisy) and provides interpretability. The second level, conditioned on the first, is a multivariate Gaussian variable, which characterizes specific attribute configurations (e.g. noise level, speaking rate) and enables disentangled fine-grained control over these attributes. This amounts to using a Gaussian mixture model (GMM) for the latent distribution. Extensive evaluation demonstrates its ability to control the aforementioned attributes. In particular, we train a high-quality controllable TTS model on real found data, which is capable of inferring speaker and style attributes from a noisy utterance and use it to synthesize clean speech with controllable speaking style.
The authors describe the conditioned GAN model to generate speaker conditioned Mel spectra. They augment the z-space corresponding to the identification with latent variables that allow a richer set of produced audio. In a way this is like a partially conditioned model that has "extra" degrees of freedom. It looks that the "latent" variables are just concaneted to the "original" set of z-values (altough with particular conditions to maximize independence). The conditioning of the z-space has originality in it and may provide interesting to the audience. Ultimately one coud think about z-space direction being totally mapped to specific features of the produced signal.
Also, I am curious to know how the Mel spectra are used to produce the actual sound wave - as the phase information is not present if utilizing only the spectral amplitude. Very often this leads to suboptimal generation, and the remedy is to use the time domain like in ( https://arxiv.org/ftp/arxiv/papers/1810/1810.05319.pdf).  However, in this case the audio samples show a pretty nice generation of sound.  However, it is not really end to end.
The manuscript has some curious decisios in its concepts - I do not see the architecture really hiearchial, nor end to end. I would prefer modifications on the paper that concentrate on the truly novel features. 
The paper is clear, well written and done with high ambition, from data set utilization  to novel architetures to human quality panels. Results are good and interesting.
NEW:
The authors have addressed the concerns I had with the manuscript.