SAMPLE EFFICIENT ADAPTIVE TEXT-TO-SPEECH
We present a meta-learning approach for adaptive text-to-speech (TTS) with few data. During training, we learn a multi-speaker model using a shared conditional WaveNet core and independent learned embeddings for each speaker. The aim of training is not to produce a neural network with fixed weights, which is then deployed as a TTS system. Instead, the aim is to produce a network that requires few data at deployment time to rapidly adapt to new speakers. We introduce and benchmark three strategies: (i) learning the speaker embedding while keeping the WaveNet core fixed, (ii) fine-tuning the entire architecture with stochastic gradient descent, and (iii) predicting the speaker embedding with a trained neural network encoder. The experiments show that these approaches are successful at adapting the multi-speaker neural network to new speakers, obtaining state-of-the-art results in both sample naturalness and voice similarity with merely a few minutes of audio data from new speakers.
This paper proposes  an adaptation technique for TTS using wavenet as the speech backend, with the adaptation carried out on small data. The work is extremely significant in that speech data is hard to produce  (we need many hours of speaker data), and techniques to adapt (transfer learning?) data from large networks would be quite valuable. The main idea is that we train a network containing a large amount of data, and (assuming that we have a trained model), we adapt this network to the task of generating speech from text for a much smaller dataset. 
In general (insofar as we can use that term), one trains such a network using <text/speech> pairs,  with speaker conditioning as added input so as to produce voice from a given speaker. The input text is converted to linguistic features in the ‘front end’, which is then injected with voice features to be synthesized into a voice output in the backend. More recent efforts in speech modeling have used RNN or wavenet based systems to carry out these transformations in the front/backends. The present work seems to use an SPSS technique (Zen et al 2016) to generate the linguistic features, while the task of converting to voice is carried out by a Wavenet. 
The work is quite (conceptually) similar to "Neural voice cloning with a few samples" (Arik et al, https://arxiv.org/abs/1802.06006) in proposing techniques for few shot adaptation described below but with the significant difference that the latter used autoregressive DNNs (loosely speaking, seq2seq a la Tacotron) for the task in both the front and back ends, while in the current work, the linguistic features are computed with SPSS as in Zel et al (2016).
The paper proposes three quite related techniques for adaptation as shown clearly in Figure 2 of the paper. These techniques are again ‘roughly’ analogous to those described in the Baidu work “Neural Voice Cloning with a few samples”, with the difference in front and backend setups noted in the previous paragraph.
We take as text as input, and convert them to a representation for linguistic features as described in Zen et al (2016). To this, we now add the fundamental frequency F_0 for the sample voice. The key piece needed is the speaker embeddings (a vector), which is to be obtained by training. In addition to all this, we also have available the weights of a trained wavenet network (probably quite large) trained on many speakers, which we will modify (or not) using the strategies outlined for the few data dataset.
SEA-EMB - Train embeddings, but not the network. We expect this to be ‘fast’, but not particularly accurate. 
SEA-ALL - Train embeddings, and network. This would be a much more accurate, if slower task. The authors note that since we train a very large network in this case, it could be prone to overfitting. They employ early stopping (as a practitioner, I would make note of the issue) with 10 % of the dataset being held out. Additional ideas such as initializing the emeddings - possibly with those that SEA-EMB calculates - are also stated to be useful.
SEA-ENC - In this third version, they predict speaker embeddings from the trained larger network (the recipe is provided in the appendix). This task of predicting speaker embeddings is one of training a classifier.
Results
The paper presents evaluations conducted with subjective, MOS based enrolment and with an evaluation metric from TI-SV d-vectors. Comparisons are made for all three models with human evaluated MOS scores, and it is seen that SEA-ALL outperforms the other two models, while performance in SEA-EMB depends on the amount of data used. Nevertheless, humans are still able to detect the difference between synthetic voices and real samples. 
The TI-SV evaluations from Wan et al show t-SNE embeddings of ‘clusters’ of d-vectors for human and synthetic voices, where it is seen that inter-cluster distance (i.e. between different speakers) is high, showing that the model is able to discern speakers, and the intra-cluster distance (i.e. between real and synthetic voices) is low, showing that synthetic voices are ‘similar’ to real voices. In addition, three other measures - cosine similarity, and statistical measures for detection error trade off, ROC curves and cosine similarity measures are also presented, which show that that the adaptation models perform quite well. 
Clarifications and comments:
Have there been efforts to compare this model (with the SPSS based frontend) with seq2seq (Bahdanau/transformer) DNN based systems as in “Neural Voice cloning with few samples”?. How do they compare (is it even a valid comparison?)?
I think the model for computing linguistic features could be elaborated upon further. 
Representations: I assume that the output audio representation is an audio waveform
Typo 1 (minor): The reference  for “Bornschein et al” in section 4 “Related work”
“Variable inference for memory addressing”. 
Correction “Variational memory addressing in generative models”
Typo 2 (minor): Figure 6: Lower curve indicate that the verification system is having a harder time distinguishing real from generated samples. 
Correction (minor): Lower curve “indicates” ...
Summary
-------------
In summary, I am in favor of accepting this paper as it proposes a solution to adapt a trained network to one with has limited number of samples. A big issue in speech modeling is that datasets are tiny, and it is difficult to obtain good quality data at reasonable cost. It would be extremely useful to have a trained network that we can adapt for our own experiments. The related paper by Arik et al (Neural Voice cloning with a few samples) also operates with similar strategic aims, but uses a a different methodology using attention based DNNs. The paper under review should be a good addition to the toolbox of few shot adaptation/transfer learning for speech with much potential for practical use.