Background:

utilization (from nvidia-smi) is: utilization = time that at least one process is running on GPU / total sampled time * 100 . It has nothing to do with how much the process running on GPU uses GPU’s resources.
inference only ( with no data loading, since we start with all the weights on the GPU and input is merely a string of words )
tacotron2 link 22

does not peak. stays at 30% the entire time.
even if the inference takes longer than 1.5 seconds, which is longer than total sampled time, utilization is at 30%.
CentOS7 with Pytorch 0.4.1