Solved by Praveen_Srinivasan in post #7 
              
              
                It turns out the issue was torch.backends.cudnn. benchmark=True - the model has variable length tensors which led to a slowdown. With that set to false, the issue was resolved.