In this work, we first propose ParaNet, a non-autoregressive seq2seq model that converts text to spectrogram. It is fully convolutional and obtains 46.7× speed-up over Deep Voice 3 (Ping et al., 2018b) at synthesis while maintaining comparable speech quality using a WaveNet vocoder. ParaNet also produces stable alignment between text and speech on the challenging test sentences by iteratively improving the attention in a layer-by-layer manner. Based on ParaNet, we build the first fully parallel neural text-to-speech system using parallel neural vocoders, which can synthesize speech from text through a single feed-forward pass. We investigate several parallel vocoders within the TTS system, including variants of IAF vocoders and bipartite flow vocoder.
This paper proposes improvements to the TTS architecture in DeepVoice3. 
For the most part (as far as I am aware), neural architectures for TTS are encoder-decoder based. The backbone can either be an RNN or a wavenet style model (autoregressive). Synthesizing audio from the feature representation (mel spectrogram) by a neural vocoder is usually an autoregressive model from the wavenet family. 
In the current work, the main novelties seem to be the following:
1) Replace the autoregressive component in the decoder (seemingly inspired by DeepVoice3) with a non-autoregressive model. This could be very advantageous because synthesis in autoregressive models can be quite slow owing to the sample level generations (and to overcome this defect, faster sampling with inverse autoregressive flows has been used in parallel wavenet, clarinet, etc. and probability density distillation). Here, if I can interpret the paper correctly, the authors use a trained attention model (autoregressive), to distill attention for the non-autoregressive setup used, which would otherwise have difficulties learning alignment. The paper claims speed up of ~50X over DeepVoice 3 which is very significant.
2) In the mel to audio converter (vocoder), the proposal is a VAE Wavenet, with appropriate modifications for the sequence modeling. The authors mention that there are similarities to the approach used in Clarinet (this has a closed form KLD between the distilled distributions, which makes things easy). 
Experiments: It is mentioned that poor attention alignments are the cause of many issues in these architectures (repeats, mispronunciations, skipped words, etc.), and they go on to show that their architecture fares well as regards these metrics, while maintaining a comparable MOS score with DeepVoice3+Wavenet. 
Evaluations: I am reasonably convinced by the audio demos provided. 
My thoughts: 
The paper is generally a good addition to the TTS literature. These are difficult to implement, brittle setups, and a practitioner could spend a lot of time debugging their broken attention curves. To that end, I feel that any technique that throws light into the modeling process would be useful for the practitioner. It is suggested that we can use a non-autoregressive model with speedups. Likewise, they also use a wavenet VAE, which they claim can be trained without distillation (could the authors please clarify this point?)
I do, however, feel that the paper has a few drawbacks. 
1) The presentation is not at all clear. This is a very subjective comment, but I feel that this work might be unreadable to someone who hasn't studied the literature (starting from Tacotron, DeepVoice 1, 2, 3, wavenet, parallel wavenet, transformer, distillation, etc.). Furthermore, the paper does not seem to make things any clearer with succinct architecture diagrams. Just as a comparison, I would like to draw attention to Tacotron (1, 2) in which I think the details can actually be worked out with some effort. 
2) Why are we using the WaveVAE - is this just to do away with the probability distillation in wavenet? Are we also using the  IAF setup as described in Kingma's work?
3) I really feel that we need more architecture diagrams for this work to be useful. That being said, the authors do provide a list of hyperparameters for potential use. 
4) Distilling attention seems to require a previously trained autoregressive attention model. What if we don't have one ready at hand?
In summary, while I see that the work will definitely be useful to the Speech Synthesis practitioner, the clarity of the paper could be improved and we need a few more diagrams (maybe even code) to make it implementable.