NEURAL TTS STYLIZATION WITH ADVERSARIAL AND COLLABORATIVE GAMES
The modeling of style when synthesizing natural human speech from text has been the focus of significant attention. Some state-of-the-art approaches train an encoder-decoder network on paired text and audio samples x txt , x aud by encouraging its output to reconstruct x aud . The synthesized audio waveform is expected to contain the verbal content of x txt and the auditory style of x aud . Unfortunately, modeling style in TTS is somewhat under-determined and training models with a reconstruction loss alone is insufficient to disentangle content and style from other factors of variation. In this work, we introduce an end-to-end TTS model that offers enhanced content-style disentanglement ability and controllability. We achieve this by combining a pairwise training procedure, an adversarial game, and a collaborative game into one training scheme. The adversarial game concentrates the true data distribution, and the collaborative game minimizes the distance between real samples and generated samples in both the original space and the latent space. As a result, our model delivers a highly controllable generator with disentangled representation. Benefiting from the separate modeling of style and content, our model can generate human fidelity speech that satisfies the desired style conditions. Our model achieves start-of-the-art results across multiple tasks, including style transfer (content and style swapping), emotion modeling, and identity transfer (fitting a new speaker's voice).
This paper proposes a method to synthesize speech from text input, with the style of an input voice provided with the text. Thus, we provide content - text - and style - voice. It leverages recent - phenomenal - progress in TTS with Deep Neural Networks as seen from exemplar works such as Tacotron (and derivatives), DeepVoice, which use seq2seq RNNs and Wavenet families of models. The work is extremely relevant in that audio data is hard to generate (expensive) and content-style modeling could be useful in a number of practical areas in synthetic voice generation. It is also quite applicable in the related problem of voice conversion. The work also uses some quite complex - (and very interesting!) - proposals to abstract style, and paste with content using generative modeling. I am VERY excited by this effort in that it puts together a number of sophisticated pieces together, in what I think is a very sensible way to implement a solution to this very difficult problem. However, I would like clarifications and explanations, especially in regards to the architecture.  
Description of problem: The paper proposes a fairly elaborate setup to inject voice style (speech) into text. At train time it takes in text samples $x_{txt}$, paired voice samples (utterances that have $x_{txt}$ as content) $s+$ and unpaired voice samples $s-$, and produces two voice samples $x+$ (for paired  <txt, utterance>) and $x-$ (for unpaired txt/utterance). The idea is that at test time, we pass in a text sample $x_{txt}$ and an UNPAIRED voice sample $x_{aud}$ and the setup produces voice in the style of $x_{aud}$ but whose content is $x_{txt}$, in other words it generates synthetic speech saying $x_{txt}$. The paper goes on to show performance metrics based on an autoencoder loss, WER and t-SNE embeddings for various attributes. 
Context:  The setup seems to be built upon the earlier work by Taigman et al (2016) which has the extremely interesting conception of using a {\it ternary} discriminator loss to carry out domain adaptation between images. This previous work was prior to the seminal CycleGAN work for image translation, which many speech works have since used. Interestingly, the Taigman work also hints at a 'common' latent representation a la UNIT using coupled VAE-GANs with cycle consistency (also extremely pertinent), but done differently. In addition to the GAN framework by Taigman et al, since this work is built upon Tacotron and the GST (Global Style Tokens) work that followed it, the generative setup is a sophisticated recurrent attention based seq2seq model.
Formulation:
A conditional formulation is used wherein the content c (encoding generated by text) is passed along with other inputs in the generator and discriminator. The formulation in Taigman assumes that there is an invariant representation in both (image) domains with shared features. To this, style embeddings (audio) gets added on and then gets passed into the generator to generate the speech. Both c and s seem to be encoder outputs in the formulation. The loss components of what they call ‘adversarial’, ‘collaborative’ and ‘style’ losses. 
Adversarial losses
The ternary loss for D consists of 
Discriminator output from ‘paired’ style embedding (i.e. text matching the content of paired audio sample)
Discriminator output from ‘unpaired’ style embedding (i.e text paired with random sample of some style)
Discriminator output from target ground truth style. The paper uses x_+, so I would think that it uses the paired sample (i.e. from the source) style.
Generator loss (also analogous to Taigman et al) consists of generations from paired and unpaired audio, possibly a loose analogue to source and target domains, although in this case we can’t as such think of ‘+’ as the source domain, since the input is text. 
Collaborative losses 
This has two components, one for style (Gatys et al 2016) and a reconstruction component. The reconstruction component again has two terms, one to reconstruct the paired audio output ‘x+=x_audio+’ - so that the input content is reproduced -  and the other to encourage reconstruction of the latent code. 
Datasets and Results:
They use two datasets: one, an internal ‘EMT-4’ dataset with 20k+ English speakers, and the other, the VCTK corpus. Comparisons are made with a few good baselines in Tacotron2, GST and DeepVoice2. 
One comparison technique to test disentanglement ability is to compare autoencoder reconstructions with the idea that a setup that has learnt to disentangle would produce higher reconstruction error because it has learnt to separate style and content. 
t-SNE embeddings are presented to show visualizations of various emotion styles (neutral, angry, sad and happy), and separation of male and female voices. A WER metric is also presented so that generations are passed into a classifier (an ASR system trained on Wavenet). All the metrics above seem to compare excellently (better than?) with the others. 
Questions and clarifications:
(Minor) There’s a typo in page 2, line 2. x_{aud}^+ should be x_{aud}^-.
Clarification on formulation: Making the analogy (is that even the right way of looking at this?) that the ‘source’ domain is ‘+’, and the target domain is ‘-’, in equation (5), the last term of the ternary discriminator has the source domain (x_{aud}^+) in it, while the Taigman et al paper uses the target term. Does this matter? I would think ‘no’, because we have a large number of terms here and each individual term in and of itself might not be relevant, nor is the current work a direct translation of the Taigman et al work. Nevertheless, I would like clarification, if possible, on the discrepancy and why we use the ‘+’ samples. 
Clarification on reconstruction loss: I think the way it is presented, equation (8) is misleading. Apparently, we are sampling from the latent space of style and content embeddings for paired data. The notation seems to be quite consistent with that of the VAE, where we have a reconstruction and a recognition model, and in effect the equation (8) is sampling from the latent space in a stochastic way. However, as far as I can see, the latent space here produces deterministic embeddings, in that c = f(x_{txt}) and s = g(x_{aud}^+), with the distribution itself being a delta function. Also, the notation q used in this equation most definitely indicates a variational distribution, which I would think is misleading (unless I have misinterpreted what the style tokens mean). At any rate, it would help to show how the style token is computed and why it is not deterministic. 
Clarification on latent reconstruction loss: In equation (9), how is the latent representation ‘l’ computed? While I can intuitively see that the latent space ‘l’ (or z, in more common notation) would be the ‘same’ between real audio samples and the ‘+’, ‘-’ fake samples, it seems to me that they would be related to s (as the paper says, ‘C’ and ‘Enc_s’ share all conv layers) and the text. But what, in physical terms is it producing? Is it like the shared latent space in the UNIT work, or the invariant representation in Taigman? This could be made clearer with an block diagram for the architecture. 
(Major) Clarification on network architecture
The work references Tacotron’s GST work (Wang et al 2018) and the related Skerry-Ryan work as the stem architecture with separate networks for style embeddings and for content (text). While the architecture itself might be available in the stem work by Wang et al, I think we need some diagrams for the current work as well for a high level picture. Although it is mentioned in words in section 3.3, I do not get a clear idea of what the encoder/decoder architectures look like. I was also surprised in not seeing attention plots which are ubiquitous in this kind of work. Furthermore, in the notes to the ‘inference’ network ‘C’ it is stated that C and Enc_s share all conv layers. Again, a diagram might be helpful - this also applies for the discriminator. 
Clarification on stability/mode collapse: Could the authors clarify how easily this setup trained in this adversarial setup? 
Note on latent representation: To put the above points in perspective, a small note on what this architecture does in regards to the meaning of the latent codes would be useful. The Taigman et al 2016 paper talks about the f-constancy condition (and 'invariance'). Likewise, in the UNIT paper by Ming-Yu Liu - which is basically a set of coupled VAEs + cycle consistency losses, there is the notion of a shared latent space. A little discussion on these aspects would make the paper much more insightful to the domain adaptation practitioner.
Reference: This reference - Adversarial feature matching for text generation - (https://arxiv.org/abs/1706.03850) contains a reconstruction stream (as perhaps many other papers) and might be useful for instruction. 
Other relevant works in speech and voice conversion: This work comes to mind, using the StarGAN setup, also containing a survey of relevant approach in voice conversion. Although the current work is for TTS, I think it would be useful to include speech papers carrying out domain adaptation for other tasks.
StarGAN-VC: Non-parallel many-to-many voice conversion with star generative adversarial networks. 
https://arxiv.org/abs/1806.02169 
I would rate this paper as being acceptable if the authors clarify my concerns, and in particular, about the architecture. It is also hard to hard to assess reproducibility in a complex architecture such as this.