Thanks for you interest,
I wasn’t really able to get satisfying results, however I’ve tried quite a few variants to get a working Text-to-Speech Model. I’ve abandoned the project a while ago, but the last Version I was trying didn’t use a GAN at all, I’ve actually settled on an Attention-Based LSTM with an Encoder + Decoder LSTM Network. But even then the attention was not really really learning to shift focus correctly, the model was suffering from overfitting. Trainnig Samples sounded very natural, but Validation Tests were very robotic, however you could slightly understand what it was saying. Back then I didn’t bother to go furhter as I was lacking enough knowledge.
If you are interessted in training a TTS Model, I recommend just checking out the existing Models like Tacotron+Waveglow or ForwardTacotron+WaveRNN/MelGAN, they sound pretty good.
If you still want to make your own TTS Model, I would highly recommend to split it up into a Mel-Spectogramm-Generator and a Vocoder as it makes it a lot easier to convert from short sequences like text to long sequences like Audio Samples.
Good luck with your project.
Kind Regards,
Florian