A 2019 Guide to Speech Synthesis with Deep Learning
I&#x27;m an author on a few of these papers referenced (the Deep Voice papers from Baidu). I&#x27;m happy to answer any questions folks may have about neural speech synthesis, as I&#x27;ve been working on this for several years now.<p>In general, it&#x27;s a fascinating space. There are challenges in text processing (not even mentioned in the blog), such as grapheme to phoneme conversion, part of speech detection, word sense disambiguation, text normalization, challenges in utterance-level modeling (spectrograms), and challenges in &quot;spectrogram inversion&quot; &#x2F; waveform synthesis. The NLP components of the pipeline are often overlooked but are no less important than they were a few years ago -- part of speech &#x2F; word sense is the difference between &quot;Time is a CONstruct&quot; and &quot;I&#x27;m going to conSTRUCT a tower&quot;, and is the difference between &quot;Let&#x27;s drop that bass&quot; being about a DJ or about a fish. The acoustic modeling phase (e.g. Tacotron, Deep Voice 3) works fairly well, and can produce some awesome demos with things like style tokens (&quot;GST-Tacotron&quot;), but still has a ways to go until it can encompass the full range of human inflection and emotion. At the waveform synthesis level, models like WaveRNN (with subscale modeling) and Parallel WaveNet make it possible to deploy modern waveform synthesis models, but it&#x27;s still a major issue to deploy them onto low-power devices due to compute restrictions. Overall, lots of interesting challenges to work on, and we&#x27;re making a lot of progress quite quickly; and I haven&#x27;t even started talking about voice conversion or voice cloning!