You could try to convert your PyTorch model to ONNX and then TensorRT as described in this blog post 13 for WaveGlow.
Also, you could have a look at this repository 7, which seems to be a bit older by now, so I’m not sure which operations will work and which won’t.