Solved by 0xFFFFFFFF in post #35 
              
              
                In this issue, a dev from nvidia explains why this problem is occuring. Essentially, the asnwer is: pytorch is not optimized well plus the nature of Tacotron2’s network architecture produced this low nvidia-smi utilization. It is not a bug.