Wow! As a newbie to TensorFlow (and to ML in general) this is a reaction I did not expect! 
TF.JS show and tell seems really cool. I’d be honored to be on! But, now I must ask you, you know this is a port of someone else’s model, right? I feel like I did only one tenth of the work they did to get this model working. That said, and motivated by the slowness of the previous model, I have been studying up on ML with plans to create a native TensorFlow.js model of FastSpeech2! Perhaps we could talk about that. 
In terms of cloning your voice with this, while this architecture is not designed to facilitate that, I wouldn’t rule it out completely. As I understand it, FastSpeech2 trains significantly faster than auto-regressive models such as Tacotron 2 (and I believe with less data but I’m not sure). A noisy and somewhat acceptable voice is possible from Tacotron 2 using only 10 minutes of training data, so I’m sure FastSpeech2 could do just as good if not better! I wouldn’t bet too much on it being fast enough in the browser, though, since the whole model would need to be trained at once.
The WASM compatibility error seems to be a missing implementation of Softplus, an activation function:
Error: Kernel 'Softplus' not registered for backend 'wasm'