I don’t know! If it is still of interest to you, another major challenge that I forgot to mention regarding why people often avoid synthesising waveforms is the sampling rate of the raw audio waveform. Generally, it is >= 16000 Hz which means for one second of audio you need to synthesise 16000 data points, which unless done by some parallel process is a lot of datapoints to synthesise. Therefore people often try to synthesise spectrograms and then transform into waveforms with help of some vocoders.